Self-supervised contrastive learning for EEG-based cross-subject motor imagery recognition

Objective. The extensive application of electroencephalography (EEG) in brain-computer interfaces (BCIs) can be attributed to its non-invasive nature and capability to offer high-resolution data. The acquisition of EEG signals is a straightforward process, but the datasets associated with these signals frequently exhibit data scarcity and require substantial resources for proper labeling. Furthermore, there is a significant limitation in the generalization performance of EEG models due to the substantial inter-individual variability observed in EEG signals.Approach. To address these issues, we propose a novel self-supervised contrastive learning framework for decoding motor imagery (MI) signals in cross-subject scenarios. Specifically, we design an encoder combining convolutional neural network and attention mechanism. In the contrastive learning training stage, the network undergoes training with the pretext task of data augmentation to minimize the distance between pairs of homologous transformations while simultaneously maximizing the distance between pairs of heterologous transformations. It enhances the amount of data utilized for training and improves the network's ability to extract deep features from original signals without relying on the true labels of the data.Main results. To evaluate our framework's efficacy, we conduct extensive experiments on three public MI datasets: BCI IV IIa, BCI IV IIb, and HGD datasets. The proposed method achieves cross-subject classification accuracies of 67.32%, 82.34%, and 81.13%on the three datasets, demonstrating superior performance compared to existing methods.Significance. Therefore, this method has great promise for improving the performance of cross-subject transfer learning in MI-based BCI systems.

A comparative study of stereo-dependent SSVEP targets and their impact on VR-BCI performance

Steady-state visual evoked potential brain-computer interfaces (SSVEP-BCI) have attracted significant attention due to their ease of deployment and high performance in terms of information transfer rate (ITR) and accuracy, making them a promising candidate for integration with consumer electronics devices. However, as SSVEP characteristics are directly associated with visual stimulus attributes, the influence of stereoscopic vision on SSVEP as a critical visual attribute has yet to be fully explored. Meanwhile, the promising combination of virtual reality (VR) devices and BCI applications is hampered by the significant disparity between VR environments and traditional 2D displays. This is not only due to the fact that screen-based SSVEP generally operates under static, stable conditions with simple and unvaried visual stimuli but also because conventional luminance-modulated stimuli can quickly induce visual fatigue. This study attempts to address these research gaps by designing SSVEP paradigms with stereo-related attributes and conducting a comparative analysis with the traditional 2D planar paradigm under the same VR environment. This study proposed two new paradigms: the 3D paradigm and the 3D-Blink paradigm. The 3D paradigm induces SSVEP by modulating the luminance of spherical targets, while the 3D-Blink paradigm employs modulation of the spheres' opacity instead. The results of offline 4-object selection experiments showed that the accuracy of 3D and 2D paradigm was 85.67 and 86.17% with canonical correlation analysis (CCA) and 86.17 and 91.73% with filter bank canonical correlation analysis (FBCCA), which is consistent with the reduction in the signal-to-noise ratio (SNR) of SSVEP harmonics for the 3D paradigm observed in the frequency-domain analysis. The 3D-Blink paradigm achieved 75.00% of detection accuracy and 27.02 bits/min of ITR with 0.8 seconds of stimulus time and task-related component analysis (TRCA) algorithm, demonstrating its effectiveness. These findings demonstrate that the 3D and 3D-Blink paradigms supported by VR can achieve improved user comfort and satisfactory performance, while further algorithmic optimization and feature analysis are required for the stereo-related paradigms. In conclusion, this study contributes to a deeper understanding of the impact of binocular stereoscopic vision mechanisms on SSVEP paradigms and promotes the application of SSVEP-BCI in diverse VR environments.

Electrode sharpness and insertion speed reduce tissue damage near high-density penetrating arrays

Objective. Over the past decade, neural electrodes have played a crucial role in bridging biological tissues with electronic and robotic devices. This study focuses on evaluating the optimal tip profile and insertion speed for effectively implanting Paradromics' high-density fine microwire arrays (FμA) prototypes into the primary visual cortex (V1) of mice and rats, addressing the challenges associated with the 'bed-of-nails' effect and tissue dimpling.Approach. Tissue response was assessed by investigating the impact of electrodes on the blood-brain barrier (BBB) and cellular damage, with a specific emphasis on tailored insertion strategies to minimize tissue disruption during electrode implantation.Main results.Electro-sharpened arrays demonstrated a marked reduction in cellular damage within 50μm of the electrode tip compared to blunt and angled arrays. Histological analysis revealed that slow insertion speeds led to greater BBB compromise than fast and pneumatic methods. Successful single-unit recordings validated the efficacy of the optimized electro-sharpened arrays in capturing neural activity.Significance.These findings underscore the critical role of tailored insertion strategies in minimizing tissue damage during electrode implantation, highlighting the suitability of electro-sharpened arrays for long-term implant applications. This research contributes to a deeper understanding of the complexities associated with high-channel-count microelectrode array implantation, emphasizing the importance of meticulous assessment and optimization of key parameters for effective integration and minimal tissue disruption. By elucidating the interplay between insertion parameters and tissue response, our study lays a strong foundation for the development of advanced implantable devices with a reduction in reactive gliosis and improved performance in neural recording applications.

Optimized FFNN with multichannel CSP-ICA framework of EEG signal for BCI

The electroencephalogram (EEG) of the patient is used to identify their motor intention, which is then converted into a control signal through a brain-computer interface (BCI) based on motor imagery. Whenever gathering features from EEG signals, making a BCI is difficult in part because of the enormous dimensionality of the data. Three stages make up the suggested methodology: pre-processing, extraction of features, selection, and categorization. To remove unwanted artifacts, the EEG signals are filtered by a fifth-order Butterworth multichannel band-pass filter. This decreases execution time and memory use, both of which improve system performance. Then a novel multichannel optimized CSP-ICA feature extraction technique is used to separate and eliminate non-discriminative information from discriminative information in the EEG channels. Furthermore, CSP uses the concept of an Artificial Bee Colony (ABC) algorithm to automatically identify the simultaneous global ideal frequency band and time interval combination for the extraction and classification of common spatial pattern characteristics. Finally, a Tunable optimized feed-forward neural network (FFNN) classifier is utilized to extract and categorize the temporal and frequency domain features, which employs an FFNN classifier with Tunable-Q wavelet transform. The proposed framework, therefore optimizes signal processing, enabling enhanced EEG signal classification for BCI applications. The result shows that the models that use Tunable optimized FFNN produce higher classification accuracy of more than 20% when compared to the existing models.

Corrigendum: Riemannian geometry-based metrics to measure and reinforce user performance changes during brain-computer interface user training

[This corrects the article DOI: 10.3389/fncom.2023.1108889.].

Fatigue factors and fatigue indices in SSVEP-based brain-computer interfaces: a systematic review and meta-analysis

Background

Fatigue is a serious challenge when applying a steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) in the real world. Many researchers have used quantitative indices to study the effect of visual stimuli on fatigue. According to a wide range of studies in fatigue analysis, there are contradictions and inconsistencies in the behavior of fatigue indicators.

New method

In this study, for the first time, a systematic review and meta-analysis were performed on fatigue indices and fatigue caused by stimulation paradigm. We queried three scientific search engines for studies published between 2000 and 2022. The inclusion criteria were papers investigating mental and visual fatigue from performing a visual task using electroencephalogram (EEG) signals.

Results

Attractiveness and variation are the most effective ways to reduce BCI fatigue. Therefore, zoom motion, Newton's ring motion, and cue patterns reduce fatigue. While the color of the cue could effectively reduce fatigue, its shape and background had no effect on fatigue. Additionally, the questionnaire and quantitative indicators such as frequency indices, signal-to-noise ratio (SNR), SSVEP amplitude, and multiscale entropy were utilized to assess fatigue. Meta-analysis indicated that when a person is fatigued, the spectrum amplitude of alpha, theta, and α + θ / β increase significantly, while SNR and SSVEP amplitude decrease significantly.

Conclusion

The outcomes of this study can be used to design more optimal stimulation protocols that cause less fatigue. Moreover, the level of fatigue can be quantitatively assessed with indicators without the participant's self-reports.

Influence of spatial frequency in visual stimuli for cVEP-based BCIs: evaluation of performance and user experience

Code-modulated visual evoked potentials (c-VEPs) are an innovative control signal utilized in brain-computer interfaces (BCIs) with promising performance. Prior studies on steady-state visual evoked potentials (SSVEPs) have indicated that the spatial frequency of checkerboard-like stimuli influences both performance and user experience. Spatial frequency refers to the dimensions of the individual squares comprising the visual stimulus, quantified in cycles (i.e., number of black-white squares pairs) per degree of visual angle. However, the specific effects of this parameter on c-VEP-based BCIs remain unexplored. Therefore, the objective of this study is to investigate the role of spatial frequency of checkerboard-like visual stimuli in a c-VEP-based BCI. Sixteen participants evaluated selection matrices with eight spatial frequencies: C001 (0 c/°, 1×1 squares), C002 (0.15 c/°, 2×2 squares), C004 (0.3 c/°, 4×4 squares), C008 (0.6 c/°, 8×8 squares), C016 (1.2 c/°, 16×16 squares), C032 (2.4 c/°, 32×32 squares), C064 (4.79 c/°, 64×64 squares), and C128 (9.58 c/°, 128×128 squares). These conditions were tested in an online spelling task, which consisted of 18 trials each conducted on a 3×3 command interface. In addition to accuracy and information transfer rate (ITR), subjective measures regarding comfort, ocular irritation, and satisfaction were collected. Significant differences in performance and comfort were observed based on different stimulus spatial frequencies. Although all conditions achieved mean accuracy over 95% after 2.1 s of trial duration, C016 stood out in terms user experience. The proposed condition not only achieved a mean accuracy of 96.53% and 164.54 bits/min with a trial duration of 1.05s, but also was reported to be significantly more comfortable than the traditional C001 stimulus. Since both features are key for BCI development, higher spatial frequencies than the classical black-to-white stimulus might be more adequate for c-VEP systems. Hence, we assert that the spatial frequency should be carefully considered in the development of future applications for c-VEP-based BCIs.

IMH-Net: a convolutional neural network for end-to-end EEG motor imagery classification

As the main component of Brain-computer interface (BCI) technology, the classification algorithm based on EEG has developed rapidly. The previous algorithms were often based on subject-dependent settings, resulting in BCI needing to be calibrated for new users. In this work, we propose IMH-Net, an end-to-end subject-independent model. The model first uses Inception blocks extracts the frequency domain features of the data, then further compresses the feature vectors to extract the spatial domain features, and finally learns the global information and classification through Multi-Head Attention mechanism. On the OpenBMI dataset, IMH-Net obtained 73.90 ± 13.10% accuracy and 73.09 ± 14.99% F1-score in subject-independent manner, which improved the accuracy by 1.96% compared with the comparison model. On the BCI competition IV dataset 2a, this model also achieved the highest accuracy and F1-score in subject-dependent manner. The IMH-Net model we proposed can improve the accuracy of subject-independent Motor Imagery (MI), and the robustness of the algorithm is high, which has strong practical value in the field of BCI.

Methods for motion artifact reduction in online brain-computer interface experiments: a systematic review

Brain-computer interfaces (BCIs) have emerged as a promising technology for enhancing communication between the human brain and external devices. Electroencephalography (EEG) is particularly promising in this regard because it has high temporal resolution and can be easily worn on the head in everyday life. However, motion artifacts caused by muscle activity, fasciculation, cable swings, or magnetic induction pose significant challenges in real-world BCI applications. In this paper, we present a systematic review of methods for motion artifact reduction in online BCI experiments. Using the PRISMA filter method, we conducted a comprehensive literature search on PubMed, focusing on open access publications from 1966 to 2022. We evaluated 2,333 publications based on predefined filtering rules to identify existing methods and pipelines for motion artifact reduction in EEG data. We present a lookup table of all papers that passed the defined filters, all used methods, and pipelines and compare their overall performance and suitability for online BCI experiments. We summarize suitable methods, algorithms, and concepts for motion artifact reduction in online BCI applications, highlight potential research gaps, and discuss existing community consensus. This review aims to provide a comprehensive overview of the current state of the field and guide researchers in selecting appropriate methods for motion artifact reduction in online BCI experiments.

Classifying human emotions in HRI: applying global optimization model to EEG brain signals

Significant efforts have been made in the past decade to humanize both the form and function of social robots to increase their acceptance among humans. To this end, social robots have recently been combined with brain-computer interface (BCI) systems in an attempt to give them an understanding of human mental states, particularly emotions. However, emotion recognition using BCIs poses several challenges, such as subjectivity of emotions, contextual dependency, and a lack of reliable neuro-metrics for real-time processing of emotions. Furthermore, the use of BCI systems introduces its own set of limitations, such as the bias-variance trade-off, dimensionality, and noise in the input data space. In this study, we sought to address some of these challenges by detecting human emotional states from EEG brain activity during human-robot interaction (HRI). EEG signals were collected from 10 participants who interacted with a Pepper robot that demonstrated either a positive or negative personality. Using emotion valence and arousal measures derived from frontal brain asymmetry (FBA), several machine learning models were trained to classify human's mental states in response to the robot personality. To improve classification accuracy, all proposed classifiers were subjected to a Global Optimization Model (GOM) based on feature selection and hyperparameter optimization techniques. The results showed that it is possible to classify a user's emotional responses to the robot's behavior from the EEG signals with an accuracy of up to 92%. The outcome of the current study contributes to the first level of the Theory of Mind (ToM) in Human-Robot Interaction, enabling robots to comprehend users' emotional responses and attribute mental states to them. Our work advances the field of social and assistive robotics by paving the way for the development of more empathetic and responsive HRI in the future.

SincMSNet: a Sinc filter convolutional neural network for EEG motor imagery classification

Objective.Motor imagery (MI) is widely used in brain-computer interfaces (BCIs). However, the decode of MI-EEG using convolutional neural networks (CNNs) remains a challenge due to individual variability.Approach.We propose a fully end-to-end CNN called SincMSNet to address this issue. SincMSNet employs the Sinc filter to extract subject-specific frequency band information and utilizes mixed-depth convolution to extract multi-scale temporal information for each band. It then applies a spatial convolutional block to extract spatial features and uses a temporal log-variance block to obtain classification features. The model of SincMSNet is trained under the joint supervision of cross-entropy and center loss to achieve inter-class separable and intra-class compact representations of EEG signals.Main results.We evaluated the performance of SincMSNet on the BCIC-IV-2a (four-class) and OpenBMI (two-class) datasets. SincMSNet achieves impressive results, surpassing benchmark methods. In four-class and two-class inter-session analysis, it achieves average accuracies of 80.70% and 71.50% respectively. In four-class and two-class single-session analysis, it achieves average accuracies of 84.69% and 76.99% respectively. Additionally, visualizations of the learned band-pass filter bands by Sinc filters demonstrate the network's ability to extract subject-specific frequency band information from EEG.Significance.This study highlights the potential of SincMSNet in improving the performance of MI-EEG decoding and designing more robust MI-BCIs. The source code for SincMSNet can be found at:https://github.com/Want2Vanish/SincMSNet.

BrainWave-Scattering Net: a lightweight network for EEG-based motor imagery recognition

Objective.Brain-computer interfaces (BCIs) enable a direct communication of the brain with the external world, using one's neural activity, measured by electroencephalography (EEG) signals. In recent years, convolutional neural networks (CNNs) have been widely used to perform automatic feature extraction and classification in various EEG-based tasks. However, their undeniable benefits are counterbalanced by the lack of interpretability properties as well as the inability to perform sufficiently when only limited amount of training data is available.Approach.In this work, we introduce a novel, lightweight, fully-learnable neural network architecture that relies on Gabor filters to delocalize EEG signal information into scattering decomposition paths along frequency and slow-varying temporal modulations.Main results.We utilize our network in two distinct modeling settings, for building either a generic (training across subjects) or a personalized (training within a subject) classifier.Significance.In both cases, using two different publicly available datasets and one in-house collected dataset, we demonstrate high performance for our model with considerably less number of trainable parameters as well as shorter training time compared to other state-of-the-art deep architectures. Moreover, our network demonstrates enhanced interpretability properties emerging at the level of the temporal filtering operation and enables us to train efficient personalized BCI models with limited amount of training data.

Investigating EEG-based cross-session and cross-task vigilance estimation in BCI systems

Objective. The state of vigilance is crucial for effective performance in brain-computer interface (BCI) tasks, and therefore, it is essential to investigate vigilance levels in BCI tasks. Despite this, most studies have focused on vigilance levels in driving tasks rather than on BCI tasks, and the electroencephalogram (EEG) patterns of vigilance states in different BCI tasks remain unclear. This study aimed to identify similarities and differences in EEG patterns and performances of vigilance estimation in different BCI tasks and sessions.Approach.To achieve this, we built a steady-state visual evoked potential-based BCI system and a rapid serial visual presentation-based BCI system and recruited 18 participants to carry out four BCI experimental sessions over four days.Main results. Our findings demonstrate that specific neural patterns for high and low vigilance levels are relatively stable across sessions. Differential entropy features significantly differ between different vigilance levels in all frequency bands and between BCI tasks in the delta and theta frequency bands, with the theta frequency band features playing a critical role in vigilance estimation. Additionally, prefrontal, temporal, and occipital regions are more relevant to the vigilance state in BCI tasks. Our results suggest that cross-session vigilance estimation is more accurate than cross-task estimation.Significance.Our study clarifies the underlying mechanisms of vigilance state in two BCI tasks and provides a foundation for further research in vigilance estimation in BCI applications.

Classification Algorithm for Electroencephalogram-based Motor Imagery Using Hybrid Neural Network with Spatio-temporal Convolution and Multi-head Attention Mechanism

Motor imagery (MI) is a brain-computer interface (BCI) technique in which specific brain regions are activated when people imagine their limbs (or muscles) moving, even without actual movement. The technology converts electroencephalogram (EEG) signals generated by the brain into computer-readable commands by measuring neural activity. Classification of motor imagery is one of the tasks in BCI. Researchers have done a lot of work on motor imagery classification, and the existing literature has relatively mature decoding methods for two-class motor tasks. However, as the categories of EEG-based motor imagery tasks increase, further exploration is needed for decoding research on four-class motor imagery tasks. In this study, we designed a hybrid neural network that combines spatiotemporal convolution and attention mechanisms. Specifically, the data is first processed by spatiotemporal convolution to extract features and then processed by a Multi-branch Convolution block. Finally, the processed data is input into the encoder layer of the Transformer for a self-attention calculation to obtain the classification results. Our approach was tested on the well-known MI datasets BCI Competition IV 2a and 2b, and the results show that the 2a dataset has a global average classification accuracy of 83.3% and a kappa value of 0.78. Experimental results show that the proposed method outperforms most of the existing methods.

Towards best practice of interpreting deep learning models for EEG-based brain computer interfaces

Introduction

As deep learning has achieved state-of-the-art performance for many tasks of EEG-based BCI, many efforts have been made in recent years trying to understand what have been learned by the models. This is commonly done by generating a heatmap indicating to which extent each pixel of the input contributes to the final classification for a trained model. Despite the wide use, it is not yet understood to which extent the obtained interpretation results can be trusted and how accurate they can reflect the model decisions.

Methods

We conduct studies to quantitatively evaluate seven different deep interpretation techniques across different models and datasets for EEG-based BCI.

Results

The results reveal the importance of selecting a proper interpretation technique as the initial step. In addition, we also find that the quality of the interpretation results is inconsistent for individual samples despite when a method with an overall good performance is used. Many factors, including model structure and dataset types, could potentially affect the quality of the interpretation results.

Discussion

Based on the observations, we propose a set of procedures that allow the interpretation results to be presented in an understandable and trusted way. We illustrate the usefulness of our method for EEG-based BCI with instances selected from different scenarios.

Using occipital ⍺-bursts to modulate behavior in real-time

Pre-stimulus endogenous neural activity can influence the processing of upcoming sensory input and subsequent behavioral reactions. Despite it is known that spontaneous oscillatory activity mostly appears in stochastic bursts, typical approaches based on trial averaging fail to capture this. We aimed at relating spontaneous oscillatory bursts in the alpha band (8-13 Hz) to visual detection behavior, via an electroencephalography-based brain-computer interface (BCI) that allowed for burst-triggered stimulus presentation in real-time. According to alpha theories, we hypothesized that visual targets presented during alpha-bursts should lead to slower responses and higher miss rates, whereas targets presented in the absence of bursts (low alpha activity) should lead to faster responses and higher false alarm rates. Our findings support the role of bursts of alpha oscillations in visual perception and exemplify how real-time BCI systems can be used as a test bench for brain-behavioral theories.

Survey on the research direction of EEG-based signal processing

Electroencephalography (EEG) is increasingly important in Brain-Computer Interface (BCI) systems due to its portability and simplicity. In this paper, we provide a comprehensive review of research on EEG signal processing techniques since 2021, with a focus on preprocessing, feature extraction, and classification methods. We analyzed 61 research articles retrieved from academic search engines, including CNKI, PubMed, Nature, IEEE Xplore, and Science Direct. For preprocessing, we focus on innovatively proposed preprocessing methods, channel selection, and data augmentation. Data augmentation is classified into conventional methods (sliding windows, segmentation and recombination, and noise injection) and deep learning methods [Generative Adversarial Networks (GAN) and Variation AutoEncoder (VAE)]. We also pay attention to the application of deep learning, and multi-method fusion approaches, including both conventional algorithm fusion and fusion between conventional algorithms and deep learning. Our analysis identifies 35 (57.4%), 18 (29.5%), and 37 (60.7%) studies in the directions of preprocessing, feature extraction, and classification, respectively. We find that preprocessing methods have become widely used in EEG classification (96.7% of reviewed papers) and comparative experiments have been conducted in some studies to validate preprocessing. We also discussed the adoption of channel selection and data augmentation and concluded several mentionable matters about data augmentation. Furthermore, deep learning methods have shown great promise in EEG classification, with Convolutional Neural Networks (CNNs) being the main structure of deep neural networks (92.3% of deep learning papers). We summarize and analyze several innovative neural networks, including CNNs and multi-structure fusion. However, we also identified several problems and limitations of current deep learning techniques in EEG classification, including inappropriate input, low cross-subject accuracy, unbalanced between parameters and time costs, and a lack of interpretability. Finally, we highlight the emerging trend of multi-method fusion approaches (49.2% of reviewed papers) and analyze the data and some examples. We also provide insights into some challenges of multi-method fusion. Our review lays a foundation for future studies to improve EEG classification performance.

A garment that measures brain activity: proof of concept of an EEG sensor layer fully implemented with smart textiles

This paper presents the first garment capable of measuring brain activity with accuracy comparable to that of state-of-the art dry electroencephalogram (EEG) systems. The main innovation is an EEG sensor layer (i.e., the electrodes, the signal transmission, and the cap support) made entirely of threads, fabrics, and smart textiles, eliminating the need for metal or plastic materials. The garment is connected to a mobile EEG amplifier to complete the measurement system. As a first proof of concept, the new EEG system (Garment-EEG) was characterized with respect to a state-of-the-art Ag/AgCl dry-EEG system (Dry-EEG) over the forehead area of healthy participants in terms of: (1) skin-electrode impedance; (2) EEG activity; (3) artifacts; and (4) user ergonomics and comfort. The results show that the Garment-EEG system provides comparable recordings to Dry-EEG, but it is more susceptible to artifacts under adverse recording conditions due to poorer contact impedances. The textile-based sensor layer offers superior ergonomics and comfort compared to its metal-based counterpart. We provide the datasets recorded with Garment-EEG and Dry-EEG systems, making available the first open-access dataset of an EEG sensor layer built exclusively with textile materials. Achieving user acceptance is an obstacle in the field of neurotechnology. The introduction of EEG systems encapsulated in wearables has the potential to democratize neurotechnology and non-invasive brain-computer interfaces, as they are naturally accepted by people in their daily lives. Furthermore, supporting the EEG implementation in the textile industry may result in lower cost and less-polluting manufacturing processes compared to metal and plastic industries.

Decoding and geometry of ten finger movements in human posterior parietal cortex and motor cortex

Objective. Enable neural control of individual prosthetic fingers for participants with upper-limb paralysis.Approach. Two tetraplegic participants were each implanted with a 96-channel array in the left posterior parietal cortex (PPC). One of the participants was additionally implanted with a 96-channel array near the hand knob of the left motor cortex (MC). Across tens of sessions, we recorded neural activity while the participants attempted to move individual fingers of the right hand. Offline, we classified attempted finger movements from neural firing rates using linear discriminant analysis with cross-validation. The participants then used the neural classifier online to control individual fingers of a brain-machine interface (BMI). Finally, we characterized the neural representational geometry during individual finger movements of both hands.Main Results. The two participants achieved 86% and 92% online accuracy during BMI control of the contralateral fingers (chance = 17%). Offline, a linear decoder achieved ten-finger decoding accuracies of 70% and 66% using respective PPC recordings and 75% using MC recordings (chance = 10%). In MC and in one PPC array, a factorized code linked corresponding finger movements of the contralateral and ipsilateral hands.Significance. This is the first study to decode both contralateral and ipsilateral finger movements from PPC. Online BMI control of contralateral fingers exceeded that of previous finger BMIs. PPC and MC signals can be used to control individual prosthetic fingers, which may contribute to a hand restoration strategy for people with tetraplegia.

Hyper-parameter tuning and feature extraction for asynchronous action detection from sub-thalamic nucleus local field potentials

Introduction

Decoding brain states from subcortical local field potentials (LFPs) indicative of activities such as voluntary movement, tremor, or sleep stages, holds significant potential in treating neurodegenerative disorders and offers new paradigms in brain-computer interface (BCI). Identified states can serve as control signals in coupled human-machine systems, e.g., to regulate deep brain stimulation (DBS) therapy or control prosthetic limbs. However, the behavior, performance, and efficiency of LFP decoders depend on an array of design and calibration settings encapsulated into a single set of hyper-parameters. Although methods exist to tune hyper-parameters automatically, decoders are typically found through exhaustive trial-and-error, manual search, and intuitive experience.

Methods

This study introduces a Bayesian optimization (BO) approach to hyper-parameter tuning, applicable through feature extraction, channel selection, classification, and stage transition stages of the entire decoding pipeline. The optimization method is compared with five real-time feature extraction methods paired with four classifiers to decode voluntary movement asynchronously based on LFPs recorded with DBS electrodes implanted in the subthalamic nucleus of Parkinson's disease patients.

Results

Detection performance, measured as the geometric mean between classifier specificity and sensitivity, is automatically optimized. BO demonstrates improved decoding performance from initial parameter setting across all methods. The best decoders achieve a maximum performance of 0.74 ± 0.06 (mean ± SD across all participants) sensitivity-specificity geometric mean. In addition, parameter relevance is determined using the BO surrogate models.

Discussion

Hyper-parameters tend to be sub-optimally fixed across different users rather than individually adjusted or even specifically set for a decoding task. The relevance of each parameter to the optimization problem and comparisons between algorithms can also be difficult to track with the evolution of the decoding problem. We believe that the proposed decoding pipeline and BO approach is a promising solution to such challenges surrounding hyper-parameter tuning and that the study's findings can inform future design iterations of neural decoders for adaptive DBS and BCI.

Corrigendum: Review of public motor imagery and execution datasets in brain-computer interfaces

[This corrects the article DOI: 10.3389/fnhum.2023.1134869.].

Somatosensory Event-Related Potential as an Electrophysiological Correlate of Endogenous Spatial Tactile Attention: Prospects for Electrotactile Brain-Computer Interface for Sensory Training

Tactile attention tasks are used in the diagnosis and treatment of neurological and sensory processing disorders, while somatosensory event-related potentials (ERP) measured by electroencephalography (EEG) are used as neural correlates of attention processes. Brain-computer interface (BCI) technology provides an opportunity for the training of mental task execution via providing online feedback based on ERP measures. Our recent work introduced a novel electrotactile BCI for sensory training, based on somatosensory ERP; however, no previous studies have addressed specific somatosensory ERP morphological features as measures of sustained endogenous spatial tactile attention in the context of BCI control. Here we show the morphology of somatosensory ERP responses induced by a novel task introduced within our electrotactile BCI platform i.e., the sustained endogenous spatial electrotactile attention task. By applying pulsed electrical stimuli to the two proximal stimulation hotspots at the user's forearm, stimulating sequentially the mixed branches of radial and median nerves with equal probability of stimuli occurrence, we successfully recorded somatosensory ERPs for both stimulation locations, in the attended and unattended conditions. Waveforms of somatosensory ERP responses for both mixed nerve branches showed similar morphology in line with previous reports on somatosensory ERP components obtained by stimulation of exclusively sensory nerves. Moreover, we found statistically significant increases in ERP amplitude on several components, at both stimulation hotspots, while sustained endogenous spatial electrotactile attention task is performed. Our results revealed the existence of general ERP windows of interest and signal features that can be used to detect sustained endogenous tactile attention and classify between spatial attention locations in 11 healthy subjects. The current results show that features of N140, P3a and P3b somatosensory ERP components are the most prominent global markers of sustained spatial electrotactile attention, over all subjects, within our novel electrotactile BCI task/paradigm, and this work proposes the features of those components as markers of sustained endogenous spatial tactile attention in online BCI control. Immediate implications of this work are the possible improvement of online BCI control within our novel electrotactile BCI system, while these finding can be used for other tactile BCI applications in the diagnosis and treatment of neurological disorders by employing mixed nerve somatosensory ERPs and sustained endogenous electrotactile attention task as control paradigms.

Review of public motor imagery and execution datasets in brain-computer interfaces

The demand for public datasets has increased as data-driven methodologies have been introduced in the field of brain-computer interfaces (BCIs). Indeed, many BCI datasets are available in various platforms or repositories on the web, and the studies that have employed these datasets appear to be increasing. Motor imagery is one of the significant control paradigms in the BCI field, and many datasets related to motor tasks are open to the public already. However, to the best of our knowledge, these studies have yet to investigate and evaluate the datasets, although data quality is essential for reliable results and the design of subject- or system-independent BCIs. In this study, we conducted a thorough investigation of motor imagery/execution EEG datasets recorded from healthy participants published over the past 13 years. The 25 datasets were collected from six repositories and subjected to a meta-analysis. In particular, we reviewed the specifications of the recording settings and experimental design, and evaluated the data quality measured by classification accuracy from standard algorithms such as Common Spatial Pattern (CSP) and Linear Discriminant Analysis (LDA) for comparison and compatibility across the datasets. As a result, we found that various stimulation types, such as text, figure, or arrow, were used to instruct subjects what to imagine and the length of each trial also differed, ranging from 2.5 to 29 s with a mean of 9.8 s. Typically, each trial consisted of multiple sections: pre-rest (2.38 s), imagination ready (1.64 s), imagination (4.26 s, ranging from 1 to 10 s), the post-rest (3.38 s). In a meta-analysis of the total of 861 sessions from all datasets, the mean classification accuracy of the two-class (left-hand vs. right-hand motor imagery) problem was 66.53%, and the population of the BCI poor performers, those who are unable to reach proficiency in using a BCI system, was 36.27% according to the estimated accuracy distribution. Further, we analyzed the CSP features and found that each dataset forms a cluster, and some datasets overlap in the feature space, indicating a greater similarity among them. Finally, we checked the minimal essential information (continuous signals, event type/latency, and channel information) that should be included in the datasets for convenient use, and found that only 71% of the datasets met those criteria. Our attempts to evaluate and compare the public datasets are timely, and these results will contribute to understanding the dataset's quality and recording settings as well as the use of using public datasets for future work on BCIs.

Riemannian geometry-based metrics to measure and reinforce user performance changes during brain-computer interface user training

Despite growing interest and research into brain-computer interfaces (BCI), their usage remains limited outside of research laboratories. One reason for this is BCI inefficiency, the phenomenon where a significant number of potential users are unable to produce machine-discernible brain signal patterns to control the devices. To reduce the prevalence of BCI inefficiency, some have advocated for novel user-training protocols that enable users to more effectively modulate their neural activity. Important considerations for the design of these protocols are the assessment measures that are used for evaluating user performance and for providing feedback that guides skill acquisition. Herein, we present three trial-wise adaptations (running, sliding window and weighted average) of Riemannian geometry-based user-performance metrics (classDistinct reflecting the degree of class separability and classStability reflecting the level of within-class consistency) to enable feedback to the user following each individual trial. We evaluated these metrics, along with conventional classifier feedback, using simulated and previously recorded sensorimotor rhythm-BCI data to assess their correlation with and discrimination of broader trends in user performance. Analysis revealed that the sliding window and weighted average variants of our proposed trial-wise Riemannian geometry-based metrics more accurately reflected performance changes during BCI sessions compared to conventional classifier output. The results indicate the metrics are a viable method for evaluating and tracking user performance changes during BCI-user training and, therefore, further investigation into how these metrics may be presented to users during training is warranted.

A flexible speller based on time-space frequency conversion SSVEP stimulation paradigm under dry electrode

Introduction

Speller is the best way to express the performance of the brain-computer interface (BCI) paradigm. Due to its advantages of short analysis time and high accuracy, the SSVEP paradigm has been widely used in the BCI speller system based on the wet electrode. It is widely known that the wet electrode operation is cumbersome and that the subjects have a poor experience. In addition, in the asynchronous SSVEP system based on threshold analysis, the system flickers continuously from the beginning to the end of the experiment, which leads to visual fatigue. The dry electrode has a simple operation and provides a comfortable experience for subjects. The EOG signal can avoid the stimulation of SSVEP for a long time, thus reducing fatigue.

Methods

This study first designed the brain-controlled switch based on continuous blinking EOG signal and SSVEP signal to improve the flexibility of the BCI speller. Second, in order to increase the number of speller instructions, we designed the time-space frequency conversion (TSFC) SSVEP stimulus paradigm by constantly changing the time and space frequency of SSVEP sub-stimulus blocks, and designed a speller in a dry electrode environment.

Results

Seven subjects participated and completed the experiments. The results showed that the accuracy of the brain-controlled switch designed in this study was up to 94.64%, and all the subjects could use the speller flexibly. The designed 60-character speller based on the TSFC-SSVEP stimulus paradigm has an accuracy rate of 90.18% and an information transmission rate (ITR) of 117.05 bits/min. All subjects can output the specified characters in a short time.

Discussion

This study designed and implemented a multi-instruction SSVEP speller based on dry electrode. Through the combination of EOG and SSVEP signals, the speller can be flexibly controlled. The frequency of SSVEP stimulation sub-block is recoded in time and space by TSFC-SSVEP stimulation paradigm, which greatly improves the number of output instructions of BCI system in dry electrode environment. This work only uses FBCCA algorithm to test the stimulus paradigm, which requires a long stimulus time. In the future, we will use trained algorithms to study stimulus paradigm to improve its overall performance.

Modeling the acceptability of BCIs for motor rehabilitation after stroke: A large scale study on the general public

Introduction

Strokes leave around 40% of survivors dependent in their activities of daily living, notably due to severe motor disabilities. Brain-computer interfaces (BCIs) have been shown to be efficiency for improving motor recovery after stroke, but this efficiency is still far from the level required to achieve the clinical breakthrough expected by both clinicians and patients. While technical levers of improvement have been identified (e.g., sensors and signal processing), fully optimized BCIs are pointless if patients and clinicians cannot or do not want to use them. We hypothesize that improving BCI acceptability will reduce patients' anxiety levels, while increasing their motivation and engagement in the procedure, thereby favoring learning, ultimately, and motor recovery. In other terms, acceptability could be used as a lever to improve BCI efficiency. Yet, studies on BCI based on acceptability/acceptance literature are missing. Thus, our goal was to model BCI acceptability in the context of motor rehabilitation after stroke, and to identify its determinants.

Methods

The main outcomes of this paper are the following: i) we designed the first model of acceptability of BCIs for motor rehabilitation after stroke, ii) we created a questionnaire to assess acceptability based on that model and distributed it on a sample representative of the general public in France (N = 753, this high response rate strengthens the reliability of our results), iii) we validated the structure of this model and iv) quantified the impact of the different factors on this population.

Results

Results show that BCIs are associated with high levels of acceptability in the context of motor rehabilitation after stroke and that the intention to use them in that context is mainly driven by the perceived usefulness of the system. In addition, providing people with clear information regarding BCI functioning and scientific relevance had a positive influence on acceptability factors and behavioral intention.

Discussion

With this paper we propose a basis (model) and a methodology that could be adapted in the future in order to study and compare the results obtained with: i) different stakeholders, i.e., patients and caregivers; ii) different populations of different cultures around the world; and iii) different targets, i.e., other clinical and non-clinical BCI applications.

Neuronal representation of bimanual arm motor imagery in the motor cortex of a tetraplegia human, a pilot study

Introduction

How the human brain coordinates bimanual movements is not well-established.

Methods

Here, we recorded neural signals from a paralyzed individual's left motor cortex during both unimanual and bimanual motor imagery tasks and quantified the representational interaction between arms by analyzing the tuning parameters of each neuron.

Results

We found a similar proportion of neurons preferring each arm during unimanual movements, however, when switching to bimanual movements, the proportion of contralateral preference increased to 71.8%, indicating contralateral lateralization. We also observed a decorrelation process for each arm's representation across the unimanual and bimanual tasks. We further confined that these changes in bilateral relationships are mainly caused by the alteration of tuning parameters, such as the increased bilateral preferred direction (PD) shifts and the significant suppression in bilateral modulation depths (MDs), especially the ipsilateral side.

Discussion

These results contribute to the knowledge of bimanual coordination and thus the design of cutting-edge bimanual brain-computer interfaces.

Neurorestoratology: New Advances in Clinical Therapy

Neurorestorative treatments have been able to improve the quality of life for patients suffering from neurological diseases and damages since the concept of Neurorestoratology was proposed. The discipline of Neurorestoratology focuses on restoring impaired neurological functions and/or structures through varying neurorestorative mechanisms including neurostimulation or neuromodulation, neuroprotection, neuroplasticity, neuroreplacement, loop reconstruction, remyelination, immunoregulation, angiogenesis or revascularization, neuroregeneration or neurogenesis and others. The neurorestorative strategies of Neurorestoratology include all therapeutic methods which can restore dysfunctions for patients with neurological diseases and improve their quality of life. Neurorestoratology is different from regenerative medicine in the nervous system, which mainly focuses on the neuroregeneration. It also is different from Neurorehabilitation. Neurorestoratology and Neurorehabilitation share some functional recovering mechanisms, such as neuroplasticity, especially in the early phase of neurological diseases; but generally Neurorehabilitation mainly focuses on recovering neurological functions through making the best use of residual neurological functions, replacing lost neurological functions in the largest degree, and preventing and treating varying complications. Recently, there have been more advances in restoring damaged nerves by cell therapy, neurostimulation/neuromodulation and braincomputer interface (BCI), neurorestorative surgery, neurorestorative pharmaceutics, and other clinic strategies. Simultaneously related therapeutic guidelines and standards are set up in succession. Based on those advances, clinicians should consider injured and degenerated nervous disorders or diseases in the central nervous system as treatable or neurorestorative disorders. Extending and encouraging further neurorestorative explorations and achieving better clinical efficacy with stronger evidence regarding neurorestoratology will shed new light and discover superior benefits for patients with neurological disorders.

Motor imagery electroencephalogram classification algorithm based on joint features in the spatial and frequency domains and instance transfer

Introduction

Motor imagery electroencephalography (MI-EEG) has significant application value in the field of rehabilitation, and is a research hotspot in the brain-computer interface (BCI) field. Due to the small training sample size of MI-EEG of a single subject and the large individual differences among different subjects, existing classification models have low accuracy and poor generalization ability in MI classification tasks.

Methods

To solve this problem, this paper proposes a electroencephalography (EEG) joint feature classification algorithm based on instance transfer and ensemble learning. Firstly, the source domain and target domain data are preprocessed, and then common space mode (CSP) and power spectral density (PSD) are used to extract spatial and frequency domain features respectively, which are combined into EEG joint features. Finally, an ensemble learning algorithm based on kernel mean matching (KMM) and transfer learning adaptive boosting (TrAdaBoost) is used to classify MI-EEG.

Results

To validate the effectiveness of the algorithm, this paper compared and analyzed different algorithms on the BCI Competition IV Dataset 2a, and further verified the stability and effectiveness of the algorithm on the BCI Competition IV Dataset 2b. The experimental results show that the algorithm has an average accuracy of 91.5% and 83.7% on Dataset 2a and Dataset 2b, respectively, which is significantly better than other algorithms.

Discussion

The statement explains that the algorithm fully exploits EEG signals and enriches EEG features, improves the recognition of the MI signals, and provides a new approach to solving the above problem.

Design of virtual BCI channels based on informer

The precision and reliability of electroencephalogram (EEG) data are essential for the effective functioning of a brain-computer interface (BCI). As the number of BCI acquisition channels increases, more EEG information can be gathered. However, having too many channels will reduce the practicability of the BCI system, raise the likelihood of poor-quality channels, and lead to information misinterpretation. These issues pose challenges to the advancement of BCI systems. Determining the optimal configuration of BCI acquisition channels can minimize the number of channels utilized, but it is challenging to maintain the original operating system and accommodate individual variations in channel layout. To address these concerns, this study introduces the EEG-completion-informer (EC-informer), which is based on the Informer architecture known for its effectiveness in time-series problems. By providing input from four BCI acquisition channels, the EC-informer can generate several virtual acquisition channels to extract additional EEG information for analysis. This approach allows for the direct inheritance of the original model, significantly reducing researchers' workload. Moreover, EC-informers demonstrate strong performance in damaged channel repair and poor channel identification. Using the Informer as a foundation, the study proposes the EC-informer, tailored to BCI requirements and demanding only a small number of training samples. This approach eliminates the need for extensive computing units to train an efficient, lightweight model while preserving comprehensive information about target channels. The study also confirms that the proposed model can be transferred to other operators with minimal loss, exhibiting robust applicability. The EC-informer's features enable original BCI devices to adapt to a broader range of classification algorithms and relax the operational requirements of BCI devices, which could facilitate the promotion of the use of BCI devices in daily life.

Recognition of single upper limb motor imagery tasks from EEG using multi-branch fusion convolutional neural network

Motor imagery-based brain-computer interfaces (MI-BCI) have important application values in the field of neurorehabilitation and robot control. At present, MI-BCI mostly use bilateral upper limb motor tasks, but there are relatively few studies on single upper limb MI tasks. In this work, we conducted studies on the recognition of motor imagery EEG signals of the right upper limb and proposed a multi-branch fusion convolutional neural network (MF-CNN) for learning the features of the raw EEG signals as well as the two-dimensional time-frequency maps at the same time. The dataset used in this study contained three types of motor imagery tasks: extending the arm, rotating the wrist, and grasping the object, 25 subjects were included. In the binary classification experiment between the grasping object and the arm-extending tasks, MF-CNN achieved an average classification accuracy of 78.52% and kappa value of 0.57. When all three tasks were used for classification, the accuracy and kappa value were 57.06% and 0.36, respectively. The comparison results showed that the classification performance of MF-CNN is higher than that of single CNN branch algorithms in both binary-class and three-class classification. In conclusion, MF-CNN makes full use of the time-domain and frequency-domain features of EEG, can improve the decoding accuracy of single limb motor imagery tasks, and it contributes to the application of MI-BCI in motor function rehabilitation training after stroke.

Recent Development of Neural Microelectrodes with Dual-Mode Detection

Neurons communicate through complex chemical and electrophysiological signal patterns to develop a tight information network. A physiological or pathological event cannot be explained by signal communication mode. Therefore, dual-mode electrodes can simultaneously monitor the chemical and electrophysiological signals in the brain. They have been invented as an essential tool for brain science research and brain-computer interface (BCI) to obtain more important information and capture the characteristics of the neural network. Electrochemical sensors are the most popular methods for monitoring neurochemical levels in vivo. They are combined with neural microelectrodes to record neural electrical activity. They simultaneously detect the neurochemical and electrical activity of neurons in vivo using high spatial and temporal resolutions. This paper systematically reviews the latest development of neural microelectrodes depending on electrode materials for simultaneous in vivo electrochemical sensing and electrophysiological signal recording. This includes carbon-based microelectrodes, silicon-based microelectrode arrays (MEAs), and ceramic-based MEAs, focusing on the latest progress since 2018. In addition, the structure and interface design of various types of neural microelectrodes have been comprehensively described and compared. This could be the key to simultaneously detecting electrochemical and electrophysiological signals.

Advances in P300 brain-computer interface spellers: toward paradigm design and performance evaluation

A brain-computer interface (BCI) is a non-muscular communication technology that provides an information exchange channel for our brains and external devices. During the decades, BCI has made noticeable progress and has been applied in many fields. One of the most traditional BCI applications is the BCI speller. This article primarily discusses the progress of research into P300 BCI spellers and reviews four types of P300 spellers: single-modal P300 spellers, P300 spellers based on multiple brain patterns, P300 spellers with multisensory stimuli, and P300 spellers with multiple intelligent techniques. For each type of P300 speller, we further review several representative P300 spellers, including their design principles, paradigms, algorithms, experimental performance, and corresponding advantages. We particularly emphasized the paradigm design ideas, including the overall layout, individual symbol shapes and stimulus forms. Furthermore, several important issues and research guidance for the P300 speller were identified. We hope that this review can assist researchers in learning the new ideas of these novel P300 spellers and enhance their practical application capability.

Frequency set selection for multi-frequency steady-state visual evoked potential-based brain-computer interfaces

Objective

Multi-frequency steady-state visual evoked potential (SSVEP) stimulation and decoding methods enable the representation of a large number of visual targets in brain-computer interfaces (BCIs). However, unlike traditional single-frequency SSVEP, multi-frequency SSVEP is not yet widely used. One of the key reasons is that the redundancy in the input options requires an additional selection process to define an effective set of frequencies for the interface. This study investigates systematic frequency set selection methods.

Methods

An optimization strategy based on the analysis of the frequency components in the resulting multi-frequency SSVEP is proposed, investigated and compared to existing methods, which are constructed based on the analysis of the stimulation (input) signals. We hypothesized that minimizing the occurrence of common sums in the multi-frequency SSVEP improves the performance of the interface, and that selection by pairs further increases the accuracy compared to selection by frequencies. An experiment with 12 participants was conducted to validate the hypotheses.

Results

Our results demonstrated a statistically significant improvement in decoding accuracy with the proposed optimization strategy based on multi-frequency SSVEP features compared to conventional techniques. Both hypotheses were validated by the experiments.

Conclusion

Performing selection by pairs and minimizing the number of common sums in selection by pairs are effective ways to select suitable frequency sets that improve multi-frequency SSVEP-based BCI accuracies.

Significance

This study provides guidance on frequency set selection in multi-frequency SSVEP. The proposed method in this study shows significant improvement in BCI performance (decoding accuracy) compared to existing methods in the literature.

Feature optimization based on improved novel global harmony search algorithm for motor imagery electroencephalogram classification

Background

Effectively decoding electroencephalogram (EEG) pattern for specific mental tasks is a crucial topic in the development of brain-computer interface (BCI). Extracting common spatial pattern (CSP) features from motor imagery EEG signals is often highly dependent on the selection of frequency band and time interval. Therefore, optimizing frequency band and time interval would contribute to effective feature extraction and accurate EEG decoding.

Objective

This study proposes an approach based on an improved novel global harmony search (INGHS) to optimize frequency-time parameters for effective CSP feature extraction.

Methods

The INGHS algorithm is applied to find the optimal frequency band and temporal interval. The linear discriminant analysis and support vector machine are used for EEG pattern decoding. Extensive experimental studies are conducted on three EEG datasets to assess the effectiveness of our proposed method.

Results

The average test accuracy obtained by the time-frequency parameters selected by the proposed INGHS method is slightly better than artificial bee colony (ABC) and particle swarm optimization (PSO) algorithms. Furthermore, the INGHS algorithm is superior to PSO and ABC in running time.

Conclusion

These superior experimental results demonstrate that the optimal frequency band and time interval selected by the INGHS algorithm could significantly improve the decoding accuracy compared with the traditional CSP method. This method has a potential to improve the performance of MI-based BCI systems.

EEG-based Brain-Computer Interfaces for people with Disorders of Consciousness: Features and applications. A systematic review

Background

Disorders of Consciousness (DoC) are clinical conditions following a severe acquired brain injury (ABI) characterized by absent or reduced awareness, known as coma, Vegetative State (VS)/Unresponsive Wakefulness Syndrome (VS/UWS), and Minimally Conscious State (MCS). Misdiagnosis rate between VS/UWS and MCS is attested around 40% due to the clinical and behavioral fluctuations of the patients during bedside consciousness assessments. Given the large body of evidence that some patients with DoC possess "covert" awareness, revealed by neuroimaging and neurophysiological techniques, they are candidates for intervention with brain-computer interfaces (BCIs).

Objectives

The aims of the present work are (i) to describe the characteristics of BCI systems based on electroencephalography (EEG) performed on DoC patients, in terms of control signals adopted to control the system, characteristics of the paradigm implemented, classification algorithms and applications (ii) to evaluate the performance of DoC patients with BCI.

Methods

The search was conducted on Pubmed, Web of Science, Scopus and Google Scholar. The PRISMA guidelines were followed in order to collect papers published in english, testing a BCI and including at least one DoC patient.

Results

Among the 527 papers identified with the first run of the search, 27 papers were included in the systematic review. Characteristics of the sample of participants, behavioral assessment, control signals employed to control the BCI, the classification algorithms, the characteristics of the paradigm, the applications and performance of BCI were the data extracted from the study. Control signals employed to operate the BCI were: P300 (N = 19), P300 and Steady-State Visual Evoked Potentials (SSVEP; hybrid system, N = 4), sensorimotor rhythms (SMRs; N = 5) and brain rhythms elicited by an emotional task (N = 1), while assessment, communication, prognosis, and rehabilitation were the possible applications of BCI in DoC patients.

Conclusion

Despite the BCI is a promising tool in the management of DoC patients, supporting diagnosis and prognosis evaluation, results are still preliminary, and no definitive conclusions may be drawn; even though neurophysiological methods, such as BCI, are more sensitive to covert cognition, it is suggested to adopt a multimodal approach and a repeated assessment strategy.

Decoding trajectories of imagined hand movement using electrocorticograms for brain-machine interface

OBJECTIVE

Reaching hand movement is an important motor skill actively examined in brain-computer interface (BCI). Among various components of movement analyzed is the hand's trajectory, which describes the hand's continuous positions in three-dimensional space. While a large body of studies have investigated the decoding of real movements and the reconstruction of real hand movement trajectories from neural signals, fewer studies have attempted to decode the trajectory of imagined hand movement. To develop BCI systems for patients with hand motor dysfunctions, the systems essentially require to achieve movement-free control of external devices, which is only possible through successful decoding of purely imagined hand movement.

APPROACH

To achieve this goal, this study used a machine learning technique (i.e., the variational Bayesian least square) to analyze the electrocorticogram (ECoG) of eighteen epilepsy patients obtained from when they performed movement execution (ME) and kinesthetic movement imagination (KMI) of the reach-and-grasp hand action.

MAIN RESULTS

The variational Bayesian decoding model was able to successfully predict the imagined trajectories of hand movement significantly above chance level. The Pearson's correlation coefficient between imagined and predicted trajectories was 0.3393 and 0.4936 for the KMI (KMI trials only) and MEKMI paradigm (alternating trials of ME and KMI) respectively.

SIGNIFICANCE

This study demonstrated a high accuracy of prediction for trajectories of imagined hand movement, and more importantly, higher decoding accuracy of imagined trajectories in the MEKMI paradigm than in the KMI paradigm solely.

A systematic review of research on augmentative and alternative communication brain-computer interface systems for individuals with disabilities

Augmentative and alternative communication brain-computer interface (AAC-BCI) systems are intended to offer communication access to people with severe speech and physical impairment (SSPI) without requiring volitional movement. As the field moves toward clinical implementation of AAC-BCI systems, research involving participants with SSPI is essential. Research has demonstrated variability in AAC-BCI system performance across users, and mixed results for comparisons of performance for users with and without disabilities. The aims of this systematic review were to (1) describe study, system, and participant characteristics reported in BCI research, (2) summarize the communication task performance of participants with disabilities using AAC-BCI systems, and (3) explore any differences in performance for participants with and without disabilities. Electronic databases were searched in May, 2018, and March, 2021, identifying 6065 records, of which 73 met inclusion criteria. Non-experimental study designs were common and sample sizes were typically small, with approximately half of studies involving five or fewer participants with disabilities. There was considerable variability in participant characteristics, and in how those characteristics were reported. Over 60% of studies reported an average selection accuracy ≤70% for participants with disabilities in at least one tested condition. However, some studies excluded participants who did not reach a specific system performance criterion, and others did not state whether any participants were excluded based on performance. Twenty-nine studies included participants both with and without disabilities, but few reported statistical analyses comparing performance between the two groups. Results suggest that AAC-BCI systems show promise for supporting communication for people with SSPI, but they remain ineffective for some individuals. The lack of standards in reporting outcome measures makes it difficult to synthesize data across studies. Further research is needed to demonstrate efficacy of AAC-BCI systems for people who experience SSPI of varying etiologies and severity levels, and these individuals should be included in system design and testing. Consensus in terminology and consistent participant, protocol, and performance description will facilitate the exploration of user and system characteristics that positively or negatively affect AAC-BCI use, and support innovations that will make this technology more useful to a broader group of people.

Clinical trial registration

https://www.crd.york.ac.uk/prospero/display_record.php?ID=CRD42018095345, PROSPERO: CRD42018095345.

2020 International brain-computer interface competition: A review

The brain-computer interface (BCI) has been investigated as a form of communication tool between the brain and external devices. BCIs have been extended beyond communication and control over the years. The 2020 international BCI competition aimed to provide high-quality neuroscientific data for open access that could be used to evaluate the current degree of technical advances in BCI. Although there are a variety of remaining challenges for future BCI advances, we discuss some of more recent application directions: (i) few-shot EEG learning, (ii) micro-sleep detection (iii) imagined speech decoding, (iv) cross-session classification, and (v) EEG(+ear-EEG) detection in an ambulatory environment. Not only did scientists from the BCI field compete, but scholars with a broad variety of backgrounds and nationalities participated in the competition to address these challenges. Each dataset was prepared and separated into three data that were released to the competitors in the form of training and validation sets followed by a test set. Remarkable BCI advances were identified through the 2020 competition and indicated some trends of interest to BCI researchers.

SAST-GCN: Segmentation Adaptive Spatial Temporal-Graph Convolutional Network for P3-Based Video Target Detection

Detecting video-induced P3 is crucial to building the video target detection system based on the brain-computer interface. However, studies have shown that the brain response patterns corresponding to video-induced P3 are dynamic and determined by the interaction of multiple brain regions. This paper proposes a segmentation adaptive spatial-temporal graph convolutional network (SAST-GCN) for P3-based video target detection. To make full use of the dynamic characteristics of the P3 signal data, the data is segmented according to the processing stages of the video-induced P3, and the brain network connections are constructed correspondingly. Then, the spatial-temporal feature of EEG data is extracted by adaptive spatial-temporal graph convolution to discriminate the target and non-target in the video. Especially, a style-based recalibration module is added to select feature maps with higher contributions and increase the feature extraction ability of the network. The experimental results demonstrate the superiority of our proposed model over the baseline methods. Also, the ablation experiments indicate that the segmentation of data to construct the brain connection can effectively improve the recognition performance by reflecting the dynamic connection relationship between EEG channels more accurately.

How to successfully classify EEG in motor imagery BCI: a metrological analysis of the state of the art

Objective. Processing strategies are analysed with respect to the classification of electroencephalographic signals related to brain-computer interfaces based on motor imagery. A review of literature is carried out to understand the achievements in motor imagery classification, the most promising trends, and the challenges in replicating these results. Main focus is placed on performance by means of a rigorous metrological analysis carried out in compliance with the international vocabulary of metrology. Hence, classification accuracy and its uncertainty are considered, as well as repeatability and reproducibility.Approach. The paper works included in the review concern the classification of electroencephalographic signals in motor-imagery- based brain-computer interfaces. Article search was carried out in accordance with the PRISMA standard and 89 studies were included.Main results. Statistically-based analyses show that brain-inspired approaches are increasingly proposed, and that these are particularly successful in discriminating against multiple classes. Notably, many proposals involve convolutional neural networks. Instead, classical machine learning approaches are still effective for binary classifications. Many proposals combine common spatial pattern, least absolute shrinkage and selection operator, and support vector machines. Regarding reported classification accuracies, performance above the upper quartile is in the 85 % to 100 % range for the binary case and in the 83 % to 93 % range for multi-class one. Associated uncertainties are up to 6 % while repeatability for a predetermined dataset is up to 8 %. Reproducibility assessment was instead prevented by lack of standardization in experiments.Significance. By relying on the analysed studies, the reader is guided towards the development of a successful processing strategy as a crucial part of a brain-computer interface. Moreover, it is suggested that future studies should extend these approaches on data from more subjects and with custom experiments, even by investigating online operation. This would also enable the quantification of results reproducibility.

The effect of stimulus number on the recognition accuracy and information transfer rate of SSVEP-BCI in augmented reality

OBJECTIVE

The biggest advantage of steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) lies in its large command set and high information transfer rate (ITR). Almost all current SSVEP-BCIs use a computer screen (CS) to present flickering visual stimuli, which limits its flexible use in actual scenes. Augmented reality (AR) technology provides the ability to superimpose visual stimuli on the real world, and it considerably expands the application scenarios of SSVEP-BCI. However, whether the advantages of SSVEP-BCI can be maintained when moving the visual stimuli to AR glasses is not known. This study investigated the effects of the stimulus number for SSVEP-BCI in an AR context.

APPROACH

We designed SSVEP flickering stimulation interfaces with four different numbers of stimulus targets and put them in AR glasses and a CS to display. Three common recognition algorithms were used to analyze the influence of the stimulus number and stimulation time on the recognition accuracy and ITR of AR-SSVEP and CS-SSVEP.

MAIN RESULTS

The amplitude spectrum and signal-to-noise ratio of AR-SSVEP were not significantly different from CS-SSVEP at the fundamental frequency but were significantly lower than CS-SSVEP at the second harmonic. SSVEP recognition accuracy decreased as the stimulus number increased in AR-SSVEP but not in CS-SSVEP. When the stimulus number increased, the maximum ITR of CS-SSVEP also increased, but not for AR-SSVEP. When the stimulus number was 25, the maximum ITR (142.05 bits/min) was reached at 400 ms. The importance of stimulation time in SSVEP was confirmed. When the stimulation time became longer, the recognition accuracy of both AR-SSVEP and CS-SSVEP increased. The peak value was reached at 3 s. The ITR increased first and then slowly decreased after reaching the peak value.

SIGNIFICANCE

Our study indicates that the conclusions based on CS-SSVEP cannot be simply applied to AR-SSVEP, and it is not advisable to set too many stimulus targets in the AR display device.

Decoding Continuous Kinetic Information of Grasp from Stereo-electroencephalographic (SEEG) Recordings

OBJECTIVE

Brain-computer interfaces (BCIs) have the potential to bypass damaged neural pathways and restore functionality lost due to injury or disease. Approaches to decoding kinematic information are well documented; however, the decoding of kinetic information has received less attention. Additionally, the possibility of using stereo-electroencephalography (SEEG) for kinetic decoding during hand grasping tasks is still largely unknown. Thus, the objective of this paper is to demonstrate kinetic parameter decoding using SEEG in patients performing a grasping task with two different force levels under two different ascending rates.

APPROACH

Temporal-spectral representations were studied to investigate frequency modulation under different force tasks. Then, force amplitude was decoded from SEEG recordings using multiple decoders, including a linear model, a partial least squares (PLS) model, an unscented Kalman filter, and three deep learning models (shallow convolutional neural network, deep convolutional neural network and the proposed CNN+RNN neural network).

MAIN RESULTS

The current study showed that: 1) for some channel, both low-frequency modulation (event-related desynchronization (ERD)) and high-frequency modulation (event-related synchronization (ERS)) were sustained during prolonged force holding periods; 2) continuously changing grasp force can be decoded from the SEEG signals; 3) the novel CNN+RNN deep learning model achieved the best decoding performance, with the predicted force magnitude closely aligned to the ground truth under different force amplitudes and changing rates.

SIGNIFICANCE

This work verified the possibility of decoding continuously changing grasp force using SEEG recordings. The result presented in this study demonstrated the potential of SEEG recordings for future BCI application.

"Mine Works Better": Examining the Influence of Embodiment in Virtual Reality on the Sense of Agency During a Binary Motor Imagery Task With a Brain-Computer Interface

Motor imagery-based brain-computer interfaces (MI-BCI) have been proposed as a means for stroke rehabilitation, which combined with virtual reality allows for introducing game-based interactions into rehabilitation. However, the control of the MI-BCI may be difficult to obtain and users may face poor performance which frustrates them and potentially affects their motivation to use the technology. Decreases in motivation could be reduced by increasing the users' sense of agency over the system. The aim of this study was to understand whether embodiment (ownership) of a hand depicted in virtual reality can enhance the sense of agency to reduce frustration in an MI-BCI task. Twenty-two healthy participants participated in a within-subject study where their sense of agency was compared in two different embodiment experiences: 1) avatar hand (with body), or 2) abstract blocks. Both representations closed with a similar motion for spatial congruency and popped a balloon as a result. The hand/blocks were controlled through an online MI-BCI. Each condition consisted of 30 trials of MI-activation of the avatar hand/blocks. After each condition a questionnaire probed the participants' sense of agency, ownership, and frustration. Afterwards, a semi-structured interview was performed where the participants elaborated on their ratings. Both conditions supported similar levels of MI-BCI performance. A significant correlation between ownership and agency was observed (r = 0.47, p = 0.001). As intended, the avatar hand yielded much higher ownership than the blocks. When controlling for performance, ownership increased sense of agency. In conclusion, designers of BCI-based rehabilitation applications can draw on anthropomorphic avatars for the visual mapping of the trained limb to improve ownership. While not While not reducing frustration ownership can improve perceived agency given sufficient BCI performance. In future studies the findings should be validated in stroke patients since they may perceive agency and ownership differently than able-bodied users.