[A review of researches on decoding algorithms of steady-state visual evoked potentials]

Brain-computer interface (BCI) systems based on steady-state visual evoked potential (SSVEP) have become one of the major paradigms in BCI research due to their high signal-to-noise ratio and short training time required by users. Fast and accurate decoding of SSVEP features is a crucial step in SSVEP-BCI research. However, the current researches lack a systematic overview of SSVEP decoding algorithms and analyses of the connections and differences between them, so it is difficult for researchers to choose the optimum algorithm under different situations. To address this problem, this paper focuses on the progress of SSVEP decoding algorithms in recent years and divides them into two categories-trained and non-trained-based on whether training data are needed. This paper also explains the fundamental theories and application scopes of decoding algorithms such as canonical correlation analysis (CCA), task-related component analysis (TRCA) and the extended algorithms, concludes the commonly used strategies for processing decoding algorithms, and discusses the challenges and opportunities in this field in the end.

Unsupervised Learning of Brain State Dynamics during Emotion Imagination using High-Density EEG

This study applies adaptive mixture independent component analysis (AMICA) to learn a set of ICA models, each optimized by fitting a distributional model for each identified component process while maximizing component process independence within some subsets of time points of a multi-channel EEG dataset. Here, we applied 20-model AMICA decomposition to long-duration (1-2 hr), high-density (128-channel) EEG data recorded while participants used guided imagination to imagine situations stimulating the experience of 15 specified emotions. These decompositions tended to return models identifying spatiotemporal EEG patterns or states within single emotion imagination periods. Model probability transitions reflected time-courses of EEG dynamics during emotion imagination, which varied across emotions. Transitions between models accounting for imagined "grief" and "happiness" were more abrupt and better aligned with participant reports, while transitions for imagined "contentment" extended into adjoining "relaxation" periods. The spatial distributions of brain-localizable independent component processes (ICs) were more similar within participants (across emotions) than emotions (across participants). Across participants, brain regions with differences in IC spatial distributions (i.e., dipole density) between emotion imagination versus relaxation were identified in or near the left rostrolateral prefrontal, posterior cingulate cortex, right insula, bilateral sensorimotor, premotor, and associative visual cortex. No difference in dipole density was found between positive versus negative emotions. AMICA models of changes in high-density EEG dynamics may allow data-driven insights into brain dynamics during emotional experience, possibly enabling the improved performance of EEG-based emotion decoding and advancing our understanding of emotion.

Data augmentation of SSVEPs using source aliasing matrix estimation for brain-computer interfaces

OBJECTIVE

Currently, ensemble task-related component analysis (eTRCA) and task discriminative component analysis (TDCA) are the state-of-the-art algorithms for steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs). However, training the BCIs requires multiple calibration trials. With insufficient calibration data, the accuracy of the BCI will degrade, or even become invalid with only one calibration trial. However, collecting a large amount of electroencephalography (EEG) data for calibration is a time-consuming and laborious process, which hinders the practical use of eTRCA and TDCA.

METHODS

This study proposed a novel method, namely Source Aliasing Matrix Estimation (SAME), to augment the calibration data for SSVEP-BCIs. SAME could generate artificial EEG trials with the featured SSVEPs. Its effectiveness was evaluated using two public datasets (i.e., Benchmark, BETA).

RESULTS

When combined with SAME, both eTRCA and TDCA had significantly improved performance with a limited number of calibration data. Specifically, SAME increased the average accuracy of eTRCA and TDCA by about 12% and 3%, respectively, with as few as two calibration trials. Notably, SAME enabled eTRCA and TDCA to work well with a single calibration trial, achieving an average accuracy >90% for the Benchmark dataset and >70% for the BETA dataset with 1-second EEG.

CONCLUSION

SAME is an effective method for SSVEP-BCIs to augment the calibration data, thereby significantly enhancing the performance of eTRCA and TDCA.

SIGNIFICANCE

We propose a new data-augmentation method that is compatible with the state-of-the-art algorithms of SSVEP-based BCIs. It can significantly reduce the efforts required to calibrate SSVEP-BCIs, which is promising for the development of practical BCIs.

Online Adaptation Boosts SSVEP-Based BCI Performance

OBJECTIVE

A user-friendly steady-state visual evoked potential (SSVEP)-based brain-computer interface (BCI) prefers no calibration for its target recognition algorithm, however, the existing calibration-free schemes perform still far behind their calibration-based counterparts. To tackle this issue, learning online from the subject's unlabeled data is investigated as a potential approach to boost the performance of the calibration-free SSVEP-based BCIs.

METHODS

An online adaptation scheme is developed to tune the spatial filters using the online unlabeled data from previous trials, and then developing the online adaptive canonical correlation analysis (OACCA) method.

RESULTS

A simulation study on two public SSVEP datasets (Dataset I and II) with a total of 105 subjects demonstrated that the proposed online adaptation scheme can boost the CCA's averaged information transfer rate (ITR) from 94.60 to 158.87 bits/min in Dataset I and from 85.80 to 123.91 bits/min in Dataset II. Furthermore, in our online experiment it boosted the CCA's ITR from 55.81 bits/min to 95.73 bits/min. More importantly, this online adaptation scheme can be easily combined with any spatial filtering-based algorithms to achieve online learning.

CONCLUSION

By online adaptation, the proposed OACCA performed much better than the calibration-free CCA, and comparable to the calibration-based algorithms.

SIGNIFICANCE

This work provides a general way for the SSVEP-based BCIs to learn online from unlabeled data and thus avoid calibration.

Resting-State EEG Signal for Major Depressive Disorder Detection: A Systematic Validation on a Large and Diverse Dataset

Major depressive disorder (MDD) is a global healthcare issue and one of the leading causes of disability. Machine learning combined with non-invasive electroencephalography (EEG) has recently been shown to have the potential to diagnose MDD. However, most of these studies analyzed small samples of participants recruited from a single source, raising serious concerns about the generalizability of these results in clinical practice. Thus, it has become critical to re-evaluate the efficacy of various common EEG features for MDD detection across large and diverse datasets. To address this issue, we collected resting-state EEG data from 400 participants across four medical centers and tested classification performance of four common EEG features: band power (BP), coherence, Higuchi's fractal dimension, and Katz's fractal dimension. Then, a sequential backward selection (SBS) method was used to determine the optimal subset. To overcome the large data variability due to an increased data size and multi-site EEG recordings, we introduced the conformal kernel (CK) transformation to further improve the MDD as compared with the healthy control (HC) classification performance of support vector machine (SVM). The results show that (1) coherence features account for 98% of the optimal feature subset; (2) the CK-SVM outperforms other classifiers such as K-nearest neighbors (K-NN), linear discriminant analysis (LDA), and SVM; (3) the combination of the optimal feature subset and CK-SVM achieves a high five-fold cross-validation accuracy of 91.07% on the training set (140 MDD and 140 HC) and 84.16% on the independent test set (60 MDD and 60 HC). The current results suggest that the coherence-based connectivity is a more reliable feature for achieving high and generalizable MDD detection performance in real-life clinical practice.

Near-zero phase-lag hyperscanning in a novel wireless EEG system

Objective. Hyperscanning is an emerging technology that concurrently scans the neural dynamics of multiple individuals to study interpersonal interactions. In particular, hyperscanning with electroencephalography (EEG) is increasingly popular owing to its mobility and its ability to allow studying social interactions in naturalistic settings at the millisecond scale.Approach.To align multiple EEG time series with sophisticated event markers in a single time domain, a precise and unified timestamp is required for stream synchronization. This study proposes a clock-synchronized method that uses a custom-made RJ45 cable to coordinate the sampling between wireless EEG amplifiers to prevent incorrect estimation of interbrain connectivity due to asynchronous sampling. In this method, analog-to-digital converters are driven by the same sampling clock. Additionally, two clock-synchronized amplifiers leverage additional radio frequency channels to keep the counter of their receiving dongles updated, which guarantees that binding event markers received by the dongle with the EEG time series have the correct timestamp.Main results.The results of two simulation experiments and one video gaming experiment reveal that the proposed method ensures synchronous sampling in a system with multiple EEG devices, achieving near-zero phase lag and negligible amplitude difference between the signals.Significance.According to all of the signal-similarity metrics, the suggested method is a promising option for wireless EEG hyperscanning and can be utilized to precisely assess the interbrain couplings underlying social-interaction behaviors.

Affective response to volitional input perturbations in obstacle avoidance and target tracking games

We present the use of two game-like tasks, Catnip and Dinorun, to explore affective responses to volitional control perturbations. We analyze behavioral and physiological measures with the self-assessment manikin (SAM), pupillometry, and electroencephalography (EEG) responses to provide intratrial emotional state as well as inter-trial correlates with selfreported survey responses. We find that subject gameplay characteristics significantly correlate with valence and dominance scores for both games, and that perturbations to the games produce a measurable decrease in response scores for Dinorun. During perturbation events, pupillometry analysis reveals considerable SAM-agnostic dilation, with stronger responses in more rigid trialized event structures. Furthermore, analyses of neural activity from central and parietal regions demonstrate significant measurable evoked responses to perturbed events across the majority of subjects for both games. By introducing perturbations, this set of experiments and analyses inform and enable further studies of affective responses to the loss of volitional control during engaging, game-like tasks.

Improving Transfer Performance of Deep Learning with Adaptive Batch Normalization for Brain-computer Interfaces. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021;2021:5800-5803. [PMID: 34892438 DOI: 10.1109/embc46164.2021.9629529] [Citation(s) in RCA: 0] [Impact Index Per Article: 0]

Recently, transfer learning and deep learning have been introduced to solve intra- and inter-subject variability problems in Brain-Computer Interfaces. However, the generalization ability of these BCIs is still to be further verified in a cross-dataset scenario. This study compared the transfer performance of manifold embedded knowledge transfer and pre-trained EEGNet with three preprocessing strategies. This study also introduced AdaBN for target domain adaptation. The results showed that EEGNet with Riemannian alignment and AdaBN could achieve the best transfer accuracy about 65.6% on the target dataset. This study may provide new insights into the design of transfer neural networks for BCIs by separating source and target batch normalization layers in the domain adaptation process.

Classification of left-versus right-hand motor imagery in stroke patients using supplementary data generated by CycleGAN

Acquiring Electroencephalography (EEG) data is often time-consuming, laborious, and costly, posing practical challenges to train powerful but data-demanding deep learning models. This study proposes a surrogate EEG data-generation system based on cycle-consistent adversarial networks (CycleGAN) that can expand the number of training data. This study used EEG2Image based on a modified S-transform (MST) to convert EEG data into EEG-topography. This method retains the frequency-domain characteristics and spatial information of the EEG signals. Then, the CycleGAN is used to learn and generate motor-imagery EEG data of stroke patients. From the visual inspection, there is no difference between the EEG topographies of the generated and original EEG data collected from the stroke patients. Finally, we used convolutional neural networks (CNN) to evaluate and analyze the generated EEG data. The experimental results show that the generated data effectively improved the classification accuracy.

Mindfulness Training Associated With Resting-State Electroencephalograms Dynamics in Novice Practitioners via Mindful Breathing and Body-Scan

Objectives: Mindfulness-based stress reduction has been proven to improve mental health and quality of life. This study examined how mindfulness training and various types of mindfulness practices altered brain activity. Methods: Specifically, the spectral powers of scalp electroencephalography of the mindfulness-based stress reduction (MBSR) group (n=17) who underwent an 8-week MBSR training-including mindful breathing and body-scan-were evaluated and compared with those of the waitlist controls (n=14). Results: Empirical results indicated that the post-intervention effect of MBSR significantly elevated the resting-state beta powers and reduced resting-state delta powers in both practices; such changes were not observed in the waitlist control. Compared with mindful breathing, body-scanning resulted in an overall decline in electroencephalograms (EEG) spectral powers at both delta and low-gamma bands among trained participants. Conclusion: Together with our preliminary data of expert mediators, the aforementioned spectral changes were salient after intervention, but mitigated along with expertise. Additionally, after receiving training, the MBSR group's mindfulness and emotion regulation levels improved significantly, which were correlated with the EEG spectral changes in the theta, alpha, and low-beta bands. The results supported that MBSR might function as a unique internal processing tool that involves increased vigilant capability and induces alterations similar to other cognitive training.

Correction to: Review of brain encoding and decoding mechanisms for EEG-based brain-computer interface

[This corrects the article DOI: 10.1007/s11571-021-09676-z.].

EEG-Based Brain-Computer Interfaces (BCIs): A Survey of Recent Studies on Signal Sensing Technologies and Computational Intelligence Approaches and Their Applications

Brain-Computer interfaces (BCIs) enhance the capability of human brain activities to interact with the environment. Recent advancements in technology and machine learning algorithms have increased interest in electroencephalographic (EEG)-based BCI applications. EEG-based intelligent BCI systems can facilitate continuous monitoring of fluctuations in human cognitive states under monotonous tasks, which is both beneficial for people in need of healthcare support and general researchers in different domain areas. In this review, we survey the recent literature on EEG signal sensing technologies and computational intelligence approaches in BCI applications, compensating for the gaps in the systematic summary of the past five years. Specifically, we first review the current status of BCI and signal sensing technologies for collecting reliable EEG signals. Then, we demonstrate state-of-the-art computational intelligence techniques, including fuzzy models and transfer learning in machine learning and deep learning algorithms, to detect, monitor, and maintain human cognitive states and task performance in prevalent applications. Finally, we present a couple of innovative BCI-inspired healthcare applications and discuss future research directions in EEG-based BCI research.

Enhancing transfer performance across datasets for brain-computer interfaces using a combination of alignment strategies and adaptive batch normalization

Objective. Recently, transfer learning (TL) and deep learning (DL) have been introduced to solve intra- and inter-subject variability problems in brain-computer interfaces (BCIs). However, current TL and DL algorithms are usually validated within a single dataset, assuming that data of the test subjects are acquired under the same condition as that of training (source) subjects. This assumption is generally violated in practice because of different acquisition systems and experimental settings across studies and datasets. Thus, the generalization ability of these algorithms needs further validations in a cross-dataset scenario, which is closer to the actual situation. This study compared the transfer performance of pre-trained deep-learning models with different preprocessing strategies in a cross-dataset scenario.Approach. This study used four publicly available motor imagery datasets, each was successively selected as a source dataset, and the others were used as target datasets. EEGNet and ShallowConvNet with four preprocessing strategies, namely channel normalization, trial normalization, Euclidean alignment, and Riemannian alignment, were trained with the source dataset. The transfer performance of pre-trained models was validated on the target datasets. This study also used adaptive batch normalization (AdaBN) for reducing interval covariate shift across datasets. This study compared the transfer performance of using the four preprocessing strategies and that of a baseline approach based on manifold embedded knowledge transfer (MEKT). This study also explored the possibility and performance of fusing MEKT and EEGNet.Main results. The results show that DL models with alignment strategies had significantly better transfer performance than the other two preprocessing strategies. As an unsupervised domain adaptation method, AdaBN could also significantly improve the transfer performance of DL models. The transfer performance of DL models that combined AdaBN and alignment strategies significantly outperformed MEKT. Moreover, the generalizability of EEGNet models that combined AdaBN and alignment strategies could be further improved via the domain adaptation step in MEKT, achieving the best generalization ability among multiple datasets (BNCI2014001: 0.788, PhysionetMI: 0.679, Weibo2014: 0.753, Cho2017: 0.650).Significance. The combination of alignment strategies and AdaBN could easily improve the generalizability of DL models without fine-tuning. This study may provide new insights into the design of transfer neural networks for BCIs by separating source and target batch normalization layers in the domain adaptation process.

Review of brain encoding and decoding mechanisms for EEG-based brain-computer interface

A brain-computer interface (BCI) can connect humans and machines directly and has achieved successful applications in the past few decades. Many new BCI paradigms and algorithms have been developed in recent years. Therefore, it is necessary to review new progress in BCIs. This paper summarizes progress for EEG-based BCIs from the perspective of encoding paradigms and decoding algorithms, which are two key elements of BCI systems. Encoding paradigms are grouped by their underlying neural meachanisms, namely sensory- and motor-related, vision-related, cognition-related and hybrid paradigms. Decoding algorithms are reviewed in four categories, namely decomposition algorithms, Riemannian geometry, deep learning and transfer learning. This review will provide a comprehensive overview of both modern primary paradigms and algorithms, making it helpful for those who are developing BCI systems.

Comparing the Differences in Brain Activities and Neural Comodulations Associated With Motion Sickness Between Drivers and Passengers

It is common to believe that passengers are more adversely affected by motion sickness than drivers. However, no study has compared passengers and drivers' neural activities and drivers experiencing motion sickness (MS). Therefore, this study attempts to explore brain dynamics in motion sickness among passengers and drivers. Eighteen volunteers participated in simulating the driving winding road experiment while their subjective motion sickness levels and electroencephalogram (EEG) signals were simultaneously recorded. Independent Component Analysis (ICA) was employed to isolate MS-related independent components (ICs) from EEG. Furthermore, comodulation analysis was applied to decompose spectra of interest ICs, related to MS, to find the specific spectra-related temporally independent modulators (IMs). The results showed that passengers' alpha band (8-12 Hz) power increased in correlation with the MS level in the parietal, occipital midline and left and right motor areas, and drivers' alpha band (8-12 Hz) power showed relatively smaller increases than those in the passenger. Further, the results also indicate that the enhanced activation of alpha IMs in the passenger than the driver is due to a higher degree of motion sickness. In conclusion, compared to the driver, the passenger experience more conflicts among multimodal sensory systems and demand neuro-physiological regulation.

Detection of fixation points using a small visual landmark for brain-computer interfaces

Objective.The speed of visual brain-computer interfaces (v-BCIs) has been greatly improved in recent years. However, the traditional v-BCI paradigms require users to directly gaze at the intensive flickering items, which would cause severe problems such as visual fatigue and excessive visual resource consumption in practical applications. Therefore, it is imperative to develop a user-friendly v-BCI.Approach.According to the retina-cortical relationship, this study developed a novel BCI paradigm to detect the fixation point of eyes using a small visual stimulus that subtended only 0.6° in visual angle and was out of the central visual field. Specifically, the visual stimulus was treated as a landmark to judge the eccentricity and polar angle of the fixation point. Sixteen different fixation points were selected around the visual landmark, i.e. different combinations of two eccentricities (2° and 4°) and eight polar angles (0,π4,π2,3π4,π,5π4,3π2and7π4). Twelve subjects participated in this study, and they were asked to gaze at one out of the 16 points for each trial. A multi-class discriminative canonical pattern matching (Multi-DCPM) algorithm was proposed to decode the user's fixation point.Main results.We found the visual stimulation landmark elicited different spatial event-related potential patterns for different fixation points. Multi-DCPM could achieve an average accuracy of 66.2% with a standard deviation of 15.8% for the classification of the sixteen fixation points, which was significantly higher than traditional algorithms (p⩽0.001). Experimental results of this study demonstrate the feasibility of using a small visual stimulus as a landmark to track the relative position of the fixation point.Significance.The proposed new paradigm provides a potential approach to alleviate the problem of irritating stimuli in v-BCIs, which can broaden the applications of BCIs.

[Classification algorithms of error-related potentials in brain-computer interface]

Error self-detection based on error-related potentials (ErrP) is promising to improve the practicability of brain-computer interface systems. But the single trial recognition of ErrP is still a challenge that hinters the development of this technology. To assess the performance of different algorithms on decoding ErrP, this paper test four kinds of linear discriminant analysis algorithms, two kinds of support vector machines, logistic regression, and discriminative canonical pattern matching (DCPM) on two open accessed datasets. All algorithms were evaluated by their classification accuracies and their generalization ability on different sizes of training sets. The study results show that DCPM has the best performance. This study shows a comprehensive comparison of different algorithms on ErrP classification, which could give guidance for the selection of ErrP algorithm.

Enhancement for P300-speller classification using multi-window discriminative canonical pattern matching

Objective.P300s are one of the most studied event-related potentials (ERPs), which have been widely used for brain-computer interfaces (BCIs). Thus, fast and accurate recognition of P300s is an important issue for BCI study. Recently, there emerges a lot of novel classification algorithms for P300-speller. Among them, discriminative canonical pattern matching (DCPM) has been proven to work effectively, in which discriminative spatial pattern (DSP) filter can significantly enhance the spatial features of P300s. However, the pattern of ERPs in space varies with time, which was not taken into consideration in the traditional DCPM algorithm.Approach.In this study, we developed an advanced version of DCPM, i.e. multi-window DCPM, which contained a series of time-dependent DSP filters to fine-tune the extraction of spatial ERP features. To verify its effectiveness, 25 subjects were recruited and they were asked to conduct the typical P300-speller experiment.Main results.As a result, multi-window DCPM achieved the character recognition accuracy of 91.84% with only five training characters, which was significantly better than the traditional DCPM algorithm. Furthermore, it was also compared with eight other popular methods, including SWLDA, SKLDA, STDA, BLDA, xDAWN, HDCA, sHDCA and EEGNet. The results showed multi-window DCPM preformed the best, especially using a small calibration dataset. The proposed algorithm was applied to the BCI Controlled Robot Contest of P300 paradigm in 2019 World Robot Conference, and won the first place.Significance.These results demonstrate that multi-window DCPM is a promising method for improving the performance and enhancing the practicability of P300-speller.

Low-Dimensional Subject Representation-Based Transfer Learning in EEG Decoding

Recently, the advances in passive brain-computer interfaces (BCIs) based on electroencephalogram (EEG) have shed light on real-world neuromonitoring technologies. However, human variability in the EEG activities hinders the development of practical applications of EEG-based BCI. To tackle this problem, many transfer-learning techniques perform supervised calibration. This kind of calibration approach requires task-relevant data, which is impractical in real-life scenarios such as drowsiness during driving. This study presents a transfer-learning framework for EEG decoding based on the low-dimensional representations of subjects learned from the pre-trial EEG. Tensor decomposition was applied to the pre-trial EEG of subjects to extract the underlying characteristics in subject, spatial, and spectral domains. Then, the proposed framework assessed the characteristics to obtain the low-dimensional subject representations such that the subjects with similar brain dynamics can be identified. This method can leverage the existing data from other users, and a small number of data from a rapid, non-task, unsupervised calibration from a new user to build an accurate BCI. Our results demonstrated that, in terms of prediction accuracy, the proposed low-dimensional subject representation-based transfer learning (LDSR-TL) framework outperformed the random selection, and the Riemannian manifold approach in cognitive-state tracking, while requiring fewer training data. The results can greatly improve the practicability, and usability of EEG-based BCI in the real world.

Learning Common Time-Frequency-Spatial Patterns for Motor Imagery Classification

The common spatial patterns (CSP) algorithm is the most popular spatial filtering method applied to extract electroencephalogram (EEG) features for motor imagery (MI) based brain-computer interface (BCI) systems. The effectiveness of the CSP algorithm depends on optimal selection of the frequency band and time window from the EEG. Many algorithms have been designed to optimize frequency band selection for CSP, while few algorithms seek to optimize the time window. This study proposes a novel framework, termed common time-frequency-spatial patterns (CTFSP), to extract sparse CSP features from multi-band filtered EEG data in multiple time windows. Specifically, the whole MI period is first segmented into multiple subseries using a sliding time window approach. Then, sparse CSP features are extracted from multiple frequency bands in each time window. Finally, multiple support vector machine (SVM) classifiers with the Radial Basis Function (RBF) kernel are trained to identify the MI tasks and the voting result of these classifiers determines the final output of the BCI. This study applies the proposed CTFSP algorithm to three public EEG datasets (BCI competition III dataset IVa, BCI competition III dataset IIIa, and BCI competition IV dataset 1) to validate its effectiveness, compared against several other state-of-the-art methods. The experimental results demonstrate that the proposed algorithm is a promising candidate for improving the performance of MI-BCI systems.

Retrosplenial Segregation Reflects the Navigation Load During Ambulatory Movement

Spatial navigation is a complex cognitive process based on vestibular, proprioceptive, and visualcues that are integrated and processed by an extensive network of brain areas. The retrosplenial complex (RSC) is an integral part of coordination and translation between spatial reference frames. Previous studies have demonstrated that the RSC is active during a spatial navigation tasks. The specifics of RSC activity under various navigation loads, however, are still not characterized. This study investigated the local information processed by the RSC under various navigation load conditions manipulated by the number of turns in the physical navigation setup. The results showed that the local information processed via the RSC, which was reflected by the segregation network, was higher when the number of turns increased, suggesting that RSC activity is associated with the navigation task load. The present findings shed light on how the brain processes spatial information in a physical navigation task.

The Current Research of Combining Multi-Modal Brain-Computer Interfaces With Virtual Reality

Combing brain-computer interfaces (BCI) and virtual reality (VR) is a novel technique in the field of medical rehabilitation and game entertainment. However, the limitations of BCI such as a limited number of action commands and low accuracy hinder the widespread use of BCI-VR. Recent studies have used hybrid BCIs that combine multiple BCI paradigms and/or the multi-modal biosensors to alleviate these issues, which may become the mainstream of BCIs in the future. The main purpose of this review is to discuss the current status of multi-modal BCI-VR. This study first reviewed the development of the BCI-VR, and explored the advantages and disadvantages of incorporating eye tracking, motor capture, and myoelectric sensing into the BCI-VR system. Then, this study discussed the development trend of the multi-modal BCI-VR, hoping to provide a pathway for further research in this field.