Comparison of recognition methods for an asynchronous (un-cued) BCI system: an investigation with 40-class SSVEP dataset

Steady-state visual evoked potential (SSVEP)-based brain-computer Interface (BCI) has demonstrated the potential to manage multi-command targets to achieve high-speed communication. Recent studies on multi-class SSVEP-based BCI have focused on synchronous systems, which rely on predefined time and task indicators; thus, these systems that use passive approaches may be less suitable for practical applications. Asynchronous systems recognize the user's intention (whether or not the user is willing to use systems) from brain activity; then, after recognizing the user's willingness, they begin to operate by switching swiftly for real-time control. Consequently, various methodologies have been proposed to capture the user's intention. However, in-depth investigation of recognition methods in asynchronous BCI system is lacking. Thus, in this work, three recognition methods (power spectral density analysis, canonical correlation analysis (CCA), and support vector machine (SVM)) used widely in asynchronous SSVEP BCI systems were explored to compare their performance. Further, we categorized asynchronous systems into two approaches (1-stage and 2-stage) based upon the recognition process's design, and compared their performance. To do so, a 40-class SSVEP dataset collected from 40 subjects was introduced. Finally, we found that the CCA-based method in the 2-stage approach demonstrated statistically significantly higher performance with a sensitivity of 97.62 ± 02.06%, specificity of 76.50 ± 23.50%, and accuracy of 75.59 ± 10.09%. Thus, it is expected that the 2-stage approach together with CCA-based recognition and FB-CCA classification have good potential to be implemented in practical asynchronous SSVEP BCI systems.

High-Frequency Discrete-Interval Binary Sequence in Asynchronous C-VEP-Based BCI for Visual Fatigue Reduction

In code-modulated visual evoked potential (c-VEP) based BCI systems, flickering visual stimuli may result in visual fatigue. Thus, we introduced a discrete-interval binary sequence (DIBS) as visual stimulus modulation, with its power spectrum optimized to emphasize high-frequency components (40 Hz-60 Hz). 8 and 17 subjects participated, respectively, in offline and online experiments on a 4-target asynchronous c-VEP-based BCI system designed to realize a high positive predictive value (PPV), a low false positive rate (FPR) during idle states, and a high true positive rate (TPR) in control states, while minimizing visual fatigue level. Two visual stimuli modulations were introduced and compared: a maximum length sequence (m-sequence) and the high-frequency discrete-interval binary sequence (DIBS). The decoding algorithm was compared among the canonical correlation analysis (CCA), the task-related component analysis (TRCA), and two approaches of sub-band component weight calculation (the traditional method and the proportional method) for FBCCA and FBTRCA. In the online experiments, the average PPV, FPR and TPR achieved, respectively [Formula: see text], [Formula: see text], [Formula: see text] with m-sequence, while [Formula: see text], [Formula: see text] and [Formula: see text] with DIBS. Estimated by objective eye-related metrics and a subjective questionnaire, the visual fatigue in DIBS cases is significantly smaller than that in m-sequence cases. In this study, the feasibility of a novel modulation approach for visual fatigue reduction was proved in an asynchronous c-VEP system, while maintaining comparable performance to existing methods, which provides further insights towards enhancing this field's long-term viability and user-friendliness.

A novel brain-controlled prosthetic hand method integrating AR-SSVEP augmentation, asynchronous control, and machine vision assistance

Background and objective

The brain-computer interface (BCI) system based on steady-state visual evoked potentials (SSVEP) is expected to help disabled patients achieve alternative prosthetic hand assistance. However, the existing study still has some shortcomings in interaction aspects such as stimulus paradigm and control logic. The purpose of this study is to innovate the visual stimulus paradigm and asynchronous decoding/control strategy by integrating augmented reality technology, and propose an asynchronous pattern recognition algorithm, thereby improving the interaction logic and practical application capabilities of the prosthetic hand with the BCI system.

Methods

An asynchronous visual stimulus paradigm based on an augmented reality (AR) interface was proposed in this paper, in which there were 8 control modes, including Grasp, Put down, Pinch, Point, Fist, Palm push, Hold pen, and Initial. According to the attentional orienting characteristics of the paradigm, a novel asynchronous pattern recognition algorithm that combines center extended canonical correlation analysis and support vector machine (Center-ECCA-SVM) was proposed. Then, this study proposed an intelligent BCI system switch based on a deep learning object detection algorithm (YOLOv4) to improve the level of user interaction. Finally, two experiments were designed to test the performance of the brain-controlled prosthetic hand system and its practical performance in real scenarios.

Results

Under the AR paradigm of this study, compared with the liquid crystal display (LCD) paradigm, the average SSVEP spectrum amplitude of multiple subjects increased by 17.41%, and the signal-noise ratio (SNR) increased by 3.52%. The average stimulus pattern recognition accuracy was 96.71 ± 3.91%, which was 2.62% higher than the LCD paradigm. Under the data analysis time of 2s, the Center-ECCA-SVM classifier obtained 94.66 ± 3.87% and 97.40 ± 2.78% asynchronous pattern recognition accuracy under the Normal metric and the Tolerant metric, respectively. And the YOLOv4-tiny model achieves a speed of 25.29fps and a 96.4% confidence in the prosthetic hand in real-time detection. Finally, the brain-controlled prosthetic hand helped the subjects to complete 4 kinds of daily life tasks in the real scene, and the time-consuming were all within an acceptable range, which verified the effectiveness and practicability of the system.

Conclusion

This research is based on improving the user interaction level of the prosthetic hand with the BCI system, and has made improvements in the SSVEP paradigm, asynchronous pattern recognition, interaction, and control logic. Furthermore, it also provides support for BCI areas for alternative prosthetic control, and movement disorder rehabilitation programs.

Spatio-temporal equalization multi-window algorithm for asynchronous SSVEP-based BCI

Objective.Asynchronous brain-computer interfaces (BCIs) show significant advantages in many practical application scenarios. Compared with the rapid development of synchronous BCIs technology, the progress of asynchronous BCI research, in terms of containing multiple targets and training-free detection, is still relatively slow. In order to improve the practicability of the BCI, a spatio-temporal equalization multi-window algorithm (STE-MW) was proposed for asynchronous detection of steady-state visual evoked potential (SSVEP) without the need for acquiring calibration data.Approach.The algorithm used SIE strategy to intercept EEG signals of different lengths through multiple stacked time windows and statistical decisions-making based on Bayesian risk decision-making. Different from the traditional asynchronous algorithms based on the 'non-control state detection' methods, this algorithm was based on the 'statistical inspection-rejection decision' mode and did not require a separate classification of non-control states, so it can be effectively applied to detections for large-scale candidates.Main results.Online experimental results involving 14 healthy subjects showed that, in the continuously input experiments of 40 targets, the algorithm achieved the average recognition accuracy of97.2±2.6%and the average information transfer rate (ITR) of106.3±32.0 bitsmin-1. At the same time, the average false alarm rate in the 240 s resting state test was0.607±0.602 min-1. In the free spelling experiments involving patients with severe amyotrophic lateral sclerosis, the subjects achieved an accuracy of 92.7% and an average ITR of 43.65 bits min^-1in two free spelling experiments.Significance.This algorithm can achieve high-performance, high-precision, and asynchronous detection of SSVEP signals with low algorithm complexity and false alarm rate under multi-targets and training-free conditions, which is helpful for the development of asynchronous BCI systems.

Signal processing algorithms for SSVEP-based brain computer interface: State-of-the-art and recent developments

Over past two decades, steady-state evoked potentials (SSVEP)-based brain computer interface (BCI) systems have been extensively developed. As we all know, signal processing algorithms play an important role in this BCI. However, there is no comprehensive review of the latest development of signal processing algorithms for SSVEP-based BCI. By analyzing the papers published in authoritative journals in nearly five years, signal processing algorithms of preprocessing, feature extraction and classification modules are discussed in detail. In addition, other aspects existed in this BCI are mentioned. The following key problems are solved. (1) In recent years, which signal processing algorithms are frequently used in each module? (2) Which signal processing algorithms attract more attention in recent years? (3) Which modules are the key to signal processing in BCI field? This information is very important for choosing the appropriate algorithms, and can also be considered as a reference for further research. Simultaneously, we hope that this work can provide relevant BCI researchers with valuable information about the latest trends of signal processing algorithms for SSVEP-based BCI systems.

An Introductory Tutorial on Brain–Computer Interfaces and Their Applications

The prospect and potentiality of interfacing minds with machines has long captured human imagination. Recent advances in biomedical engineering, computer science, and neuroscience are making brain–computer interfaces a reality, paving the way to restoring and potentially augmenting human physical and mental capabilities. Applications of brain–computer interfaces are being explored in applications as diverse as security, lie detection, alertness monitoring, gaming, education, art, and human cognition augmentation. The present tutorial aims to survey the principal features and challenges of brain–computer interfaces (such as reliable acquisition of brain signals, filtering and processing of the acquired brainwaves, ethical and legal issues related to brain–computer interface (BCI), data privacy, and performance assessment) with special emphasis to biomedical engineering and automation engineering applications. The content of this paper is aimed at students, researchers, and practitioners to glimpse the multifaceted world of brain–computer interfacing.

A Maximum Likelihood Perspective of Spatial Filter Design in SSVEP-Based BCIs

In steady-state visual-evoked potential (SSVEP) based brain-computer interfaces (BCIs), existing detection algorithms utilizing spatial filters like task-related component analysis (TRCA) derive the spatial filters mainly through maximizing the inter-trial similarity between the combined signals over the training set. Although they achieve by far the best classification performance in SSVEP-based BCIs, some important problems are still unresolved. Especially, the mechanism of how spatial filters cancel the background noise in brain signals and optimize the signal-to-noise ratio (SNR) of SSVEPs is still not figured out. Therefore, to solve these problems, in this paper a new perspective of spatial filter design is proposed. Specifically, a linear generative signal model of SSVEP is adopted and the spatial filters are obtained automatically through maximum likelihood estimation of source signals and channel vectors. In the same time, the relation between maximum likelihood estimation and signal-to-noise ratio (SNR) maximization is discussed. Through a step-by-step formulation, this paper provides a theoretical justification for those conventional algorithms utilizing spatial filters. As for the classification performance, the proposed scheme is tested on a benchmark dataset of 35 subjects. Experiment results show that the classification performance of the proposed scheme is competitive against three benchmark algorithms, which include TRCA. Especially, the proposed scheme achieves a fair performance improvement over the benchmark methods in the cases where a shorter time window, or a larger number of electrodes, or a smaller number of training blocks are adopted.

Asynchronous c-VEP communication tools-efficiency comparison of low-target, multi-target and dictionary-assisted BCI spellers

Keyboards and smartphones allow users to express their thoughts freely via manual control. Hands-free communication can be realized with brain-computer interfaces (BCIs) based on code-modulated visual evoked potentials (c-VEPs). Various variations of such spellers have been developed: Low-target systems, multi-target systems and systems with dictionary support. In general, it is not clear which kinds of systems are optimal in terms of reliability, speed, cognitive load, and visual load. The presented study investigates the feasibility of different speller variations. 58 users tested a 4-target speller and a 32-target speller with and without dictionary functionality. For classification, multiple individualized spatial filters were generated via canonical correlation analysis (CCA). We used an asynchronous implementation allowing non-control state, thus aiming for high accuracy rather than speed. All users were able to control the tested spellers. Interestingly, no significant differences in accuracy were found: 94.4%, 95.5% and 94.0% for 4-target spelling, 32-target spelling, and dictionary-assisted 32-target spelling. The mean ITRs were highest for the 32-target interface: 45.2, 96.9 and 88.9 bit/min. The output speed in characters per minute, was highest in dictionary-assisted spelling: 8.2, 19.5 and 31.6 characters/min. According to questionnaire results, 86% of the participants preferred the 32-target speller over the 4-target speller.

Graph Eigen Decomposition-Based Feature-Selection Method for Epileptic Seizure Detection Using Electroencephalography

Epileptic seizure is a sudden alteration of behavior owing to a temporary change in the electrical functioning of the brain. There is an urgent demand for an automatic epilepsy detection system using electroencephalography (EEG) for clinical application. In this paper, the EEG signal is divided into short time frames. Discrete wavelet transform is used to decompose each frame into a number of subbands. Different entropies as well as a group of features with which to characterize the spike events are extracted from each subband signal of an EEG frame. The features extracted from individual subbands are concatenated, yielding a high-dimensional feature vector. A discriminative subset of features is selected from the feature vector using a graph eigen decomposition (GED)-based approach. Thus, the reduced number of features obtained is effective for differentiating the underlying characteristics of EEG signals that indicate seizure events and those that indicate nonseizure events. The GED method ranks the features according to their contribution to correct classification. The selected features are used to classify seizure and nonseizure EEG signals using a feedforward neural network (FfNN). The performance of the proposed method is evaluated by conducting various experiments with a standard dataset obtained from the University of Bonn. The experimental results show that the proposed seizure-detection scheme achieves a classification accuracy of 99.55%, which is higher than that of state-of-the-art methods. The efficiency of FfNN is compared with linear discriminant analysis and support vector machine classifiers, which have classification accuracies of 98.72% and 99.39%, respectively. Hence, the proposed method is confirmed as a potential marker for EEG-based seizure detection.

Can a highly accurate multi-class SSMVEP BCI induce sensory-motor rhythm in sensorimotor area?

Different visual stimuli might have different effects on the brain, e.g. the change of brightness, non-biological movement and biological movement. In this study, flicker, checkerboard, and gaiting stimuli were chosen as visual stimuli to investigate whether steady-state motion visual evoked potential (SSMVEP) effected on the sensorimotor area for rehabilitation. The gaiting stimulus was designed as the gaiting sequence of a human. The hypothesis is that only observing the designed gaiting stimulus would simultaneously induce 1) SSMVEP in the occipital area, similarly to an SSVEP stimulus; and 2) sensorimotor rhythm (SMR) in the primary sensorimotor area, because such action observation could activate the mirror neuron system. Canonical correlation analysis was used to detect SSMVEP from occipital EEG, and event-related spectral perturbation was used to identify SMR in the EEG from the sensorimotor area. The results showed that the designed gaiting stimulus-induced SSMVEP, with classification accuracies of 88.9 ± 12.0% in a four-class scenario. More importantly, it induced clear and sustained event-related desynchronization/synchronization (ERD/ERS), while no ERD/ERS could be observed when the other two SSVEP stimuli were used. Further, for participants with a sufficiently clear SSMVEP pattern (classification accuracy > 85%), the ERD index values in the mu-beta band induced by the proposed gaiting stimulus were statistically different from that of the other two types of stimulus. Therefore, a novel BCI based on the designed stimulus has potential in neurorehabilitation applications because it simultaneously has the high accuracy of an SSMVEP (~90% accuracy in a four-class setup) and the ability to activate the sensorimotor area.

Spatial Filtering in SSVEP-Based BCIs: Unified Framework and New Improvements

OBJECTIVE

In the steady-state visual evoked potential (SSVEP)-based brain computer interfaces (BCIs), spatial filtering, which combines the multi-channel electroencephalography (EEG) signals in order to reduce the non-SSVEP-related component and thus enhance the signal-to-noise ratio (SNR), plays an important role in target recognition. Recently, various spatial filtering algorithms have been developed employing different prior knowledge and characteristics of SSVEPs, however how these algorithms interconnect and differ is not yet fully explored, leading to difficulties in further understanding, utilizing and improving them.

METHODS

We propose a unified framework under which the spatial filtering algorithms can be formulated as generalized eigenvalue problems (GEPs) with four different elements: data, temporal filter, orthogonal projection and spatial filter. Based on the framework, we design new spatial filtering algorithms for improvements through the choice of different elements.

RESULTS

The similarities, differences and relationships among nineteen mainstream spatial filtering algorithms are revealed under the proposed framework. Particularly, it is found that they originate from the canonical correlation analysis (CCA), principal component analysis (PCA), and multi-set CCA, respectively. Furthermore, three new spatial filtering algorithms are developed with enhanced performance validated on two public SSVEP datasets with 45 subjects.

CONCLUSION

The proposed framework provides insights into the underlying relationships among different spatial filtering algorithms and helps the design of new spatial filtering algorithms.

SIGNIFICANCE

This is a systematic study to explore, compare and improve the existing spatial filtering algorithms, which would be significant for further understanding and future development of high performance SSVEP-based BCIs.

Improve the Classification Efficiency of High-Frequency Phase-Tagged SSVEP by a Recursive Bayesian-Based Approach

Among the Steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs), the phase-tagged SSVEP (p-SSVEP) has been proved a reliable paradigm to extend the number of available targets, especially for high-frequency SSVEP-based BCIs. However, the recognition efficiency of the high-frequency p-SSVEP still remains relatively low. A longer data segment may achieve a higher classification accuracy, but the time consumption of computation leads to the decrease of information transfer rate. This paper presents a recursive Bayesian-based approach to improve the high-frequency p-SSVEP classification efficiency. In each signal processing period, the classification result is generated by the current scores, the condition probability and a recursive prior probability (dynamic prior probability). The experiment displays the SSVEP stimuli with 20 Hz and 30 Hz respectively, and each frequency contains six phases. This paper compared three classification approaches and the recursive Bayesian-based approach could obtain the highest classification accuracy and practical bit rate under the same data length. The mean accuracy and practical bit rate were 89.7% and 37.8 bits/min with 20Hz, and 89.0% and 36.5 bits/min with 30Hz, respectively Furthermore, the recursive Bayesian-based approach could reduce the individual differences among different subjects. Therefore, the recursive Bayesian-based approach can lead to high classification efficiency in high-frequency p-SSVEP.

World's fastest brain-computer interface: Combining EEG2Code with deep learning

We present a novel approach based on deep learning for decoding sensory information from non-invasively recorded Electroencephalograms (EEG). It can either be used in a passive Brain-Computer Interface (BCI) to predict properties of a visual stimulus the person is viewing, or it can be used to actively control a BCI application. Both scenarios were tested, whereby an average information transfer rate (ITR) of 701 bit/min was achieved for the passive BCI approach with the best subject achieving an online ITR of 1237 bit/min. Further, it allowed the discrimination of 500,000 different visual stimuli based on only 2 seconds of EEG data with an accuracy of up to 100%. When using the method for an asynchronous self-paced BCI for spelling, an average utility rate of 175 bit/min was achieved, which corresponds to an average of 35 error-free letters per minute. As the presented method extracts more than three times more information than the previously fastest approach, we suggest that EEG signals carry more information than generally assumed. Finally, we observed a ceiling effect such that information content in the EEG exceeds that required for BCI control, and therefore we discuss if BCI research has reached a point where the performance of non-invasive visual BCI control cannot be substantially improved anymore.

Asynchronous non-invasive high-speed BCI speller with robust non-control state detection

Brain-Computer Interfaces (BCIs) enable users to control a computer by using pure brain activity. Recent BCIs based on visual evoked potentials (VEPs) have shown to be suitable for high-speed communication. However, all recent high-speed BCIs are synchronous, which means that the system works with fixed time slots so that the user is not able to select a command at his own convenience, which poses a problem in real-world applications. In this paper, we present the first asynchronous high-speed BCI with robust distinction between intentional control (IC) and non-control (NC), with a nearly perfect NC state detection of only 0.075 erroneous classifications per minute. The resulting asynchronous speller achieved an average information transfer rate (ITR) of 122.7 bit/min using a 32 target matrix-keyboard. Since the method is based on random stimulation patterns it allows to use an arbitrary number of targets for any application purpose, which was shown by using an 55 target German QWERTZ-keyboard layout which allowed the participants to write an average of 16.1 (up to 30.7) correct case-sensitive letters per minute. As the presented system is the first asynchronous high-speed BCI speller with a robust non-control state detection, it is an important step for moving BCI applications out of the lab and into real-life.

A Convolutional Neural Network for the Detection of Asynchronous Steady State Motion Visual Evoked Potential

A key issue in brain-computer interface (BCI) is the detection of intentional control (IC) states and non-intentional control (NC) states in an asynchronous manner. Furthermore, for steady-state visual evoked potential (SSVEP) BCI systems, multiple states (sub-states) exist within the IC state. Existing recognition methods rely on a threshold technique, which is difficult to realize high accuracy, i.e., simultaneously high true positive rate and low false positive rate. To address this issue, we proposed a novel convolutional neural network (CNN) to detect IC and NC states in a SSVEP-BCI system for the first time. Specifically, the steady-state motion visual evoked potentials (SSMVEP) paradigm, which has been shown to induce less visual discomfort, was chosen as the experimental paradigm. Two processing pipelines were proposed for the detection of IC and NC states. The first one was using CNN as a multi-class classifier to discriminate between all the states in IC and NC state (FFT-CNN). The second one was using CNN to discriminate between IC and NC states, and using canonical correlation analysis (CCA) to perform classification tasks within the IC (FFT-CNN-CCA). We demonstrated that both pipelines achieved a significant increase in accuracy for low-performance healthy participants when traditional algorithms such as CCA threshold were used. Furthermore, the FFT-CNN-CCA pipeline achieved better performance than the FFT-CNN pipeline based on the stroke patients' data. In summary, we showed that CNN can be used for robust detection in an asynchronous SSMVEP-BCI with great potential for out-of-lab BCI applications.