A new hybrid BCI paradigm based on P300 and SSVEP

Application of hybrid SSVEP + P300 brain computer interface to control avatar movement in mobile virtual reality gaming environment

INTRODUCTION

This research evaluated the feasibility of a hybrid SSVEP + P300 brain computer interface (BCI) for controlling the movement of an avatar in a virtual reality (VR) gaming environment (VR + BCI). Existing VR + BCI gaming environments have limitations, such as visual fatigue, a lower communication rate, minimum accuracy, and poor system comfort. Hence, there is a need for an optimized hybrid BCI system that can simultaneously evoke the strongest P300 and SSVEP potentials in the cortex.

METHODS

A BCI headset was coupled with a VR headset to generate a VR + BCI environment. The author developed a VR game in which the avatar's movement is controlled using the user's cortical responses with the help of a BCI headset. Specifically designed visual stimuli were used in the proposed system to elicit the strongest possible responses from the user's brain. The proposed system also includes an auditory feedback mechanism to facilitate precise avatar movement.

RESULTS AND CONCLUSIONS

Conventional P300 BCI and SSVEP BCI were also used to control the movements of the avatar, and their performance metrics were compared to those of the proposed system. The results demonstrated that the hybrid SSVEP + P300 BCI system was superior to the other systems for controlling avatar movement.

The use of happy faces as visual stimuli improves the performance of the hybrid SSVEP+P300 brain computer interface

BACKGROUND

This study illustrates a hybrid brain-computer interface (BCI) in which steady-state visual evoked potentials (SSVEP) and event-related potentials (P300) are evoked simultaneously. The goal of this study was to improve the performance of the current hybrid SSVEP+P300 BCI systems by incorporating a happy face into visual stimuli.

NEW METHOD

In this study, happy and sad faces were added to a visual stimulus to induce stronger cortical signals in a hybrid SSVEP+P300 BCI. Additionally, we developed a paradigm in which SSVEP responses were triggered by non-face stimuli, whereas P300 responses were triggered by face stimuli. We tested four paradigms: happy face paradigm (HF), sad face paradigm (SF), happy face and flicker paradigm (HFF), and sad face and flicker paradigm (SFF).

RESULTS AND CONCLUSIONS

The results demonstrated that the HFF paradigm elicited more robust cortical responses, which resulted in enhanced system accuracy and information transfer rate (ITR). The HFF paradigm has a system communication rate of 25.9 bits per second and an average accuracy of 96.1%. Compared with other paradigms, the HFF paradigm is the best choice for BCI applications because it has the highest ITR and maximum level of comfort.

A delayed matching task-based study on action sequence of motor imagery

The way people imagine greatly affects performance of brain-computer interface (BCI) based on motion imagery (MI). Action sequence is a basic unit of imitation, learning, and memory for motor behavior. Whether it influences the MI-BCI is unknown, and how to manifest this influence is difficult since the MI is a spontaneous brain activity. To investigate the influence of the action sequence, this study proposes a novel paradigm named action sequences observing and delayed matching task to use images and videos to guide people to observe, match and reinforce the memory of sequence. Seven subjects' ERPs and MI performance are analyzed under four different levels of complexities or orders of the sequence. Results demonstrated that the action sequence in terms of complexity and sequence order significantly affects the MI. The complex action in positive order obtains stronger ERD/ERS and more pronounced MI feature distributions, and yields an MI classification accuracy that is 12.3% higher than complex action in negative order (p < 0.05). In addition, the ERP amplitudes derived from the supplementary motor area show a positive correlation to the MI. This study demonstrates a new perspective of improving imagery in the MI-BCI by considering the complexity and order of the action sequences, and provides a novel index for manifesting the MI performance by ERP.

Effect of inverted faces as visual stimuli on the performance of the hybrid SSVEP + P300 brain computer interface

INTRODUCTION

This study proposes a hybrid brain-computer interface (BCI) system that simultaneously evokes steady-state visual evoked potentials (SSVEP) and event-related potentials (P300). The goal of this study was to improve the performance of the current hybrid SSVEP + P300 BCI systems by incorporating inverted faces into visual stimuli.

METHODS

In this study, upright and inverted faces were added to visual stimulus to elicit stronger cortical responses in a hybrid SSVEP + P300 BCI. We also considered triggering the P300 signals with facial stimuli and the SSVEP signals with non-facial stimuli. We have tested four paradigms: the upright face paradigm (UF), the inverted face paradigm (IF), the upright face and flicker paradigm (UFF), and the inverted face and flicker paradigm (IFF).

RESULTS AND CONCLUSIONS

The results showed that the IFF paradigm evoked more robust cortical responses, which led to enhanced system accuracy and ITR. The IFF paradigm had an average accuracy of 96.6% and a system communication rate of 26.45 bits per second. The UFF paradigm is the best candidate for BCI applications among other paradigms because it provides maximum comfort while maintaining a reasonable ITR.

A Wearable Asynchronous Brain-Computer Interface Based on EEG-EOG Signals With Fewer Channels

OBJECTIVE

Brain-computer interfaces (BCIs) have tremendous application potential in communication, mechatronic control and rehabilitation. However, existing BCI systems are bulky, expensive and require laborious preparation before use. This study proposes a practical and user-friendly BCI system without compromising performance.

METHODS

A hybrid asynchronous BCI system was developed based on an elaborately designed wearable electroencephalography (EEG) amplifier that is compact, easy to use and offers a high signal-to-noise ratio (SNR). The wearable BCI system can detect P300 signals by processing EEG signals from three channels and operates asynchronously by integrating blink detection.

RESULT

The wearable EEG amplifier obtains high quality EEG signals and introduces preprocessing capabilities to BCI systems. The wearable BCI system achieves an average accuracy of 94.03±4.65%, an average information transfer rate (ITR) of 31.42±7.39 bits/min and an average false-positive rate (FPR) of 1.78%.

CONCLUSION

The experimental results demonstrate the feasibility and practicality of the developed wearable EEG amplifier and BCI system.

SIGNIFICANCE

Wearable asynchronous BCI systems with fewer channels are possible, indicating that BCI applications can be transferred from the laboratory to real-world scenarios.

Spectrum-Enhanced TRCA (SE-TRCA): A novel approach for direction detection in SSVEP-based BCI

The Steady State Visual Evoked Potential (SSVEP) is a widely used component in BCIs due to its high noise resistance and low equipment requirements. Recently, a novel SSVEP-based paradigm has been introduced for direction detection, in which, unlike the common SSVEP paradigms that use several frequency stimuli, only one flickering stimulus is used and it makes direction detection very challenging. So far, only the CCA method has been used for direction detection using SSVEP component analysis. Since Canonical Correlation Analysis (CCA) has some limitations, a Task-Related Component Analysis (TRCA) based method has been introduced for feature extraction to improve the direction detection performance. Although these methods have been proven efficient, they do not utilize the latent frequency information in the EEG signal. Therefore, the performance of direction detection using SSVEP component analysis is still suboptimal. For further improvement, the TRCA-based algorithm is enhanced by incorporating frequency information and introducing Spectrum-Enhanced TRCA (SE-TRCA). SE-TRCA method can utilize frequency information in conjunction with spatial information by concatenating the EEG signal and its shifted version. Accordingly, the obtained spatio-spectral filters perform as a Finite Impulse Response (FIR) filter. To evaluate the proposed SE-TRCA method, two different sorts of datasets (1) a hybrid BCI dataset (including SSVEP component for direction detection) and (2) a pure benchmark SSVEP dataset (including SSVEP component for frequency detection) have been used. Our experiments showed that the accuracy of direction detection using the proposed SE-TRCA and TRCA approaches compared to CCA-based approach have been increased by 23.35% and 28.24%, respectively. Furthermore, the accuracy of character recognition obtained from integrating P300 and SSVEP components in CCA, TRCA, and SETRCA approaches are 54.01%, 56.02%, and 58.56%, on the hybrid dataset, respectively. The evaluation of the SE-TRCA method on the benchmark SSVEP dataset demonstrates that the SE-TRCA method outperforms both CCA and TRCA, particularly regarding frequency detection accuracy. In this specific dataset, the SE-TRCA method achieved an impressive frequency detection accuracy of 98.19% for a 3-s signal, surpassing the accuracies of TRCA and CCA, which were 97.91% and 90.47%, respectively. These results demonstrated that the TRCA-based approach is more efficient than the CCA approach to extracting spatial filters. Moreover, SE-TRCA, extracting both Spectral and spatial information from the EEG signal, can capture more discriminative features from the SSVEP component and increase the accuracy of classification. The results of this study emphasize the effectiveness of the proposed SE-TRCA approach across different SSVEP paradigms and tasks. These findings provide strong evidence for the method's ability to generalize well in SSVEP analysis.

A hybrid P300-SSVEP brain-computer interface speller with a frequency enhanced row and column paradigm

Objective

This study proposes a new hybrid brain-computer interface (BCI) system to improve spelling accuracy and speed by stimulating P300 and steady-state visually evoked potential (SSVEP) in electroencephalography (EEG) signals.

Methods

A frequency enhanced row and column (FERC) paradigm is proposed to incorporate the frequency coding into the row and column (RC) paradigm so that the P300 and SSVEP signals can be evoked simultaneously. A flicker (white-black) with a specific frequency from 6.0 to 11.5 Hz with an interval of 0.5 Hz is assigned to one row or column of a 6 × 6 layout, and the row/column flashes are carried out in a pseudorandom sequence. A wavelet and support vector machine (SVM) combination is adopted for P300 detection, an ensemble task-related component analysis (TRCA) method is used for SSVEP detection, and the two detection possibilities are fused using a weight control approach.

Results

The implemented BCI speller achieved an accuracy of 94.29% and an information transfer rate (ITR) of 28.64 bit/min averaged across 10 subjects during the online tests. An accuracy of 96.86% is obtained during the offline calibration tests, higher than that of only using P300 (75.29%) or SSVEP (89.13%). The SVM in P300 outperformed the previous linear discrimination classifier and its variants (61.90–72.22%), and the ensemble TRCA in SSVEP outperformed the canonical correlation analysis method (73.33%).

Conclusion

The proposed hybrid FERC stimulus paradigm can improve the performance of the speller compared with the classical single stimulus paradigm. The implemented speller can achieve comparable accuracy and ITR to its state-of-the-art counterparts with advanced detection algorithms.

The steady state visual evoked potential (SSVEP) tracks "sticky" thinking, but not more general mind-wandering

For a large proportion of our daily lives, spontaneously occurring thoughts tend to disengage our minds from goal-directed thinking. Previous studies showed that EEG features such as the P3 and alpha oscillations can predict mind-wandering to some extent, but only with accuracies of around 60%. A potential candidate for improving prediction accuracy is the Steady-State Visual Evoked Potential (SSVEP), which is used frequently in single-trial contexts such as brain-computer interfaces as a marker of the direction of attention. In this study, we modified the sustained attention to response task (SART) that is usually employed to measure spontaneous thought to incorporate the SSVEP elicited by a 12.5-Hz flicker. We then examined whether the SSVEP could track and allow for the prediction of the stickiness and task-relatedness dimensions of spontaneous thought. Our results show that the SSVEP evoked by flickering words was able to distinguish between more and less sticky thinking but not between whether a participant was on- or off-task. This suggests that the SSVEP is able to track spontaneous thinking when it is strongly disengaged from the task (as in the sticky form of off-task thinking) but not off-task thought in general. Future research should determine the exact dimensions of spontaneous thought to which the SSVEP is most sensitive.

Improved classification performance of EEG-fNIRS multimodal brain-computer interface based on multi-domain features and multi-level progressive learning

Electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have potentially complementary characteristics that reflect the electrical and hemodynamic characteristics of neural responses, so EEG-fNIRS-based hybrid brain-computer interface (BCI) is the research hotspots in recent years. However, current studies lack a comprehensive systematic approach to properly fuse EEG and fNIRS data and exploit their complementary potential, which is critical for improving BCI performance. To address this issue, this study proposes a novel multimodal fusion framework based on multi-level progressive learning with multi-domain features. The framework consists of a multi-domain feature extraction process for EEG and fNIRS, a feature selection process based on atomic search optimization, and a multi-domain feature fusion process based on multi-level progressive machine learning. The proposed method was validated on EEG-fNIRS-based motor imagery (MI) and mental arithmetic (MA) tasks involving 29 subjects, and the experimental results show that multi-domain features provide better classification performance than single-domain features, and multi-modality provides better classification performance than single-modality. Furthermore, the experimental results and comparison with other methods demonstrated the effectiveness and superiority of the proposed method in EEG and fNIRS information fusion, it can achieve an average classification accuracy of 96.74% in the MI task and 98.42% in the MA task. Our proposed method may provide a general framework for future fusion processing of multimodal brain signals based on EEG-fNIRS.

A novel multiple time-frequency sequential coding strategy for hybrid brain-computer interface

Background

For brain-computer interface (BCI) communication, electroencephalography provides a preferable choice due to its high temporal resolution and portability over other neural recording techniques. However, current BCIs are unable to sufficiently use the information from time and frequency domains simultaneously. Thus, we proposed a novel hybrid time-frequency paradigm to investigate better ways of using the time and frequency information.

Method

We adopt multiple omitted stimulus potential (OSP) and steady-state motion visual evoked potential (SSMVEP) to design the hybrid paradigm. A series of pre-experiments were undertaken to study factors that would influence the feasibility of the hybrid paradigm and the interaction between multiple features. After that, a novel Multiple Time-Frequencies Sequential Coding (MTFSC) strategy was introduced and explored in experiments.

Results

Omissions with multiple short and long durations could effectively elicit time and frequency features, including the multi-OSP, ERP, and SSVEP in this hybrid paradigm. The MTFSC was feasible and efficient. The preliminary online analysis showed that the accuracy and the ITR of the nine-target stimulator over thirteen subjects were 89.04% and 36.37 bits/min.

Significance

This study first combined the SSMVEP and multi-OSP in a hybrid paradigm to produce robust and abundant time features for coding BCI. Meanwhile, the MTFSC proved feasible and showed great potential in improving performance, such as expanding the number of BCI targets by better using time information in specific stimulated frequencies. This study holds promise for designing better BCI systems with a novel coding method.

Neural decoding of semantic concepts: A systematic literature review

Objective Semantic concepts are coherent entities within our minds. They underpin our thought processes and are a part of the basis for our understanding of the world. Modern neuroscience research is increasingly exploring how individual semantic concepts are encoded within our brains and a number of studies are beginning to reveal key patterns of neural activity that underpin specific concepts. Building upon this basic understanding of the process of semantic neural encoding, neural engineers are beginning to explore tools and methods for semantic decoding: identifying which semantic concepts an individual is focused on at a given moment in time from recordings of their neural activity. In this paper we review the current literature on semantic neural decoding. Approach We conducted this review according to the Preferred Reporting Items for Systematic reviews and Meta-Analysis (PRISMA) guidelines. Specifically, we assess the eligibility of published peer-reviewed reports via a search of PubMed and Google Scholar. We identify a total of 74 studies in which semantic neural decoding is used to attempt to identify individual semantic concepts from neural activity. Results Our review reveals how modern neuroscientific tools have been developed to allow decoding of individual concepts from a range of neuroimaging modalities. We discuss specific neuroimaging methods, experimental designs, and machine learning pipelines that are employed to aid the decoding of semantic concepts. We quantify the efficacy of semantic decoders by measuring information transfer rates. We also discuss current challenges presented by this research area and present some possible solutions. Finally, we discuss some possible emerging and speculative future directions for this research area. Significance Semantic decoding is a rapidly growing area of research. However, despite its increasingly widespread popularity and use in neuroscientific research this is the first literature review focusing on this topic across neuroimaging modalities and with a focus on quantifying the efficacy of semantic decoders.

Effects of inter-stimulus intervals on concurrent P300 and SSVEP features for hybrid Brain-computer interfaces

BACKGROUND

Recently, we have implemented a high-speed brain-computer interface (BCI) system with a large instruction set using the concurrent P300 and steady-state visual evoked potential (SSVEP) features (also known as hybrid features). However, it remains unclear how to select inter-stimulus interval (ISI) for the proposed BCI system to balance the encoding efficiency and decoding performance.

NEW METHOD

This study developed a 6⁎9 hybrid P300-SSVEP BCI system and investigated a series of ISIs ranged from -175ms to 0ms with a step of 25ms. The influence of ISI on the hybrid features was analyzed from several aspects, including the amplitude of the induced features, classification accuracy, information transfer rate (ITR). Twelve naive subjects were recruited for the experiment.

RESULTS

The results showed the ISI factor had a significant impact on the hybrid features. Specifically, as the values of ISI decreased, the amplitudes of the induced features and accuracies decreased gradually, while the ITRs increased rapidly. It's achieved the highest ITR of 158.50 bits/min when ISI equal to -175ms.

COMPARISON WITH EXISTING METHOD

The optimal ISI in this study achieved superior performance in comparison with the one we used in the previous study.

CONCLUSIONS

The ISI can exert an important influence on the P300-SSVEP BCI system and its optimal value is -175ms in this study, which is significant for developing the high-speed BCI system with larger instruction sets in the future.

Improving Cross-State and Cross-Subject Visual ERP-based BCI with Temporal Modeling and Adversarial Training

Brain-computer interface (BCI) is a useful device for people without relying on peripheral nerves and muscles. However, the performance of the event-related potential (ERP)-based BCI declines when applying it to real environments, especially in cross-state and cross-subject conditions. Here we employ temporal modeling and adversarial training to improve the visual ERP-based BCI under different mental workload states and to alleviate the problems above. The rationality of our method is that the ERP-based BCI is based on electroencephalography (EEG) signals recorded from the scalp's surface, continuously changing with time and somewhat stochastic. In this paper, we propose a hierarchical recurrent network to encode all ERP signals in each repetition at the same time and model them with a temporal manner to predict which visual event elicited an ERP. The hierarchical architecture is a simple yet effective method for organizing recurrent layers in a deep structure to model long sequence signals. Taking a cue from recent advances in adversarial training, we further applied dynamic adversarial perturbations to create adversarial examples to enhance the model performance. We conduct our experiments on one published visual ERP-based BCI task with 15 subjects and 3 different auditory workload states. The results indicate that our hierarchical method can effectively model the long sequence EEG raw data, outperform the baselines on most conditions, including cross-state and cross-subject conditions. Finally, we show how deep learning-based methods with limited EEG data can improve ERP-based BCI with adversarial training. Our code will be released at https://github.com/aispeech-lab/VisBCI.

A brain-actuated robotic arm system using non-invasive hybrid brain-computer interface and shared control strategy

Objective.The electroencephalography (EEG)-based brain-computer interfaces (BCIs) have been used in the control of robotic arms. The performance of non-invasive BCIs may not be satisfactory due to the poor quality of EEG signals, so the shared control strategies were tried as an alternative solution. However, most of the existing shared control methods set the arbitration rules manually, which highly depended on the specific tasks and developer's experience. In this study, we proposed a novel shared control model that automatically optimized the control commands in a dynamical way based on the context in real-time control. Besides, we employed the hybrid BCI to better allocate commands with multiple functions. The system allowed non-invasive BCI users to manipulate a robotic arm moving in a three-dimensional (3D) space and complete a pick-place task of multiple objects.Approach.Taking the scene information obtained by computer vision as a knowledge base, a machine agent was designed to infer the user's intention and generate automatic commands. Based on the inference confidence and user's characteristic, the proposed shared control model fused the machine autonomy and human intention dynamically for robotic arm motion optimization during the online control. In addition, we introduced a hybrid BCI scheme that applied steady-state visual evoked potentials and motor imagery to the divided primary and secondary BCI interfaces to better allocate the BCI resources (e.g. decoding computing power, screen occupation) and realize the multi-dimensional control of the robotic arm.Main results.Eleven subjects participated in the online experiments of picking and placing five objects that scattered at different positions in a 3D workspace. The results showed that most of the subjects could control the robotic arm to complete accurate and robust picking task with an average success rate of approximately 85% under the shared control strategy, while the average success rate of placing task controlled by pure BCI was 50% approximately.Significance.In this paper, we proposed a novel shared controller for motion automatic optimization, together with a hybrid BCI control scheme that allocated paradigms according to the importance of commands to realize multi-dimensional and effective control of a robotic arm. Our study indicated that the shared control strategy with hybrid BCI could greatly improve the performance of the brain-actuated robotic arm system.

EEG-based hybrid QWERTY mental speller with high information transfer rate

BACKGROUND

Brain-computer interface (BCI) spellers detect variations in brain waves to help subjects communicate with the world. This study introduces a P300-SSVEP hybrid BCI-based QWERTY speller.

METHODS

The proposed hybrid speller, combines SSVEP and P300 features using a hybrid paradigm. P300 was used as time division multiplexing index which results in the use of lesser number of assumed frequencies for SSVEP elicitation. Each flickering frequency was also assigned a unique colour, to enhance system accuracy.

RESULTS

On the basis of 20 subjects, an average accuracy of classification of 96.42% and a mean information transfer rate (ITR) of 131.0 bits per min. (BPM) was achieved during the free spelling trial (trial-F).

COMPARISON

The t test results revealed that the hybrid QWERTY speller performed significantly better (on the basis of mean classification accuracy and ITR) as compared to the traditional P300 speller) and the QWERTY SSVEP speller. Also, the amount of time taken to spell a word was significantly lesser in the case of hybrid QWERTY speller in contrast to traditional P300 speller while it was almost the same as compared to QWERTY SSVEP speller.

CONCLUSION

QWERTY speller outperformed the stereotypical P300 speller as well as QWERTY SSVEP speller.

A Training Data-Driven Canonical Correlation Analysis Algorithm for Designing Spatial Filters to Enhance Performance of SSVEP-Based BCIs

Canonical correlation analysis (CCA) is an effective spatial filtering algorithm widely used in steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs). In existing CCA methods, training data are used for constructing templates of stimulus targets and the spatial filters are created between the template signals and a single-trial testing signal. The fact that spatial filters rely on testing data, however, results in low classification performance of CCA compared to other state-of-the-art algorithms such as task-related component analysis (TRCA). In this study, we proposed a novel CCA method in which spatial filters are estimated using training data only. This is achieved by using observed EEG training data and their SSVEP components as the two inputs of CCA and the objective function is optimized by averaging multiple training trials. In this case, we proved in theory that the two spatial filters estimated by the CCA are equivalent, and that the CCA and TRCA are also equivalent under certain hypotheses. A benchmark SSVEP data set from 35 subjects was used to compare the performance of the two algorithms according to different lengths of data, numbers of channels and numbers of training trials. In addition, the CCA was also compared with power spectral density analysis (PSDA). The experimental results suggest that the CCA is equivalent to TRCA if the signal-to-noise ratio of training data is high enough; otherwise, the CCA outperforms TRCA in terms of classification accuracy. The CCA is much faster than PSDA in detecting time of targets. The robustness of the training data-driven CCA to noise gives it greater potential in practical applications.

DIY hybrid SSVEP-P300 LED stimuli for BCI platform using EMOTIV EEG headset

A fully customisable chip-on board (COB) LED design to evoke two brain responses simultaneously (steady state visual evoked potential (SSVEP) and transient evoked potential, P300) is discussed in this paper. Considering different possible modalities in brain-computer interfacing (BCI), SSVEP is widely accepted as it requires a lesser number of electroencephalogram (EEG) electrodes and minimal training time. The aim of this work was to produce a hybrid BCI hardware platform to evoke SSVEP and P300 precisely with reduced fatigue and improved classification performance. The system comprises of four independent radial green visual stimuli controlled individually by a 32-bit microcontroller platform to evoke SSVEP and four red LEDs flashing at random intervals to generate P300 events. The system can also record the P300 event timestamps that can be used in classification, to improve the accuracy and reliability. The hybrid stimulus was tested for real-time classification accuracy by controlling a LEGO robot to move in four directions.

Indirect Control of an Autonomous Wheelchair Using SSVEP BCI

Having the capability to control a wheelchair using brain signals would be a major benefit to patients suffering from motor disabling diseases. However, one major challenge such systems are facing is the amount of input needed over time by the patient for control. Such a navigation control system results in a significant mental burden for the patient. The objective of this study is to develop a BCI system that requires a low number of inputs from a subject to operate. We propose an autonomous wheelchair that uses steady-state visual evoked potential based brain computer interfaces to achieve the objective. A dual mode system was implemented in this study to allow the autonomous wheelchair to work in both unknown and known environments. Robot operating system is used as the middleware in this study for the development of the algorithm to operate the wheelchair. The mental task for the subject using this wheelchair is reduced by relegating the responsibility of navigation control from the subject to the navigation software.

Asynchronous Video Target Detection Based on Single-Trial EEG Signals

Event-related potentials (ERPs) are widely used in brain-computer interface (BCI) systems to detect sensitive targets. However, asynchronous BCI systems based on video-target-evoked ERPs can pose a challenge in real-world applications due to the absence of an explicit target onset time and the time jitter of the detection latency. To address this challenge, we developed an asynchronous detection framework for video target detection. In this framework, an ERP alignment method based on the principle of iterative minimum distance square error (MDSE) was proposed for constructing an ERP template and aligning signals on the same base to compensate for possible time jitter. Using this method, ERP response characteristics induced by video targets were estimated. Online video target detection results indicated that alignment methods reduced the false alarm more effectively than non-alignment methods. The false alarm of the proposed Aligned-MDSE method was one-third lower than that of existing alignment methods under the same right hit level using limited individual samples. Furthermore, cross-subject results indicated that untrained subjects could directly perform online detection tasks and achieve excellent performance by a general model trained from more than 10 subjects. The proposed asynchronous video target detection framework can thus have a significant impact on real-world BCI applications.

A novel motion coupling coding method for brain-computer interfaces

Objectives

The best frequency response band for the steady-state visual evoked potential (SSVEP) stimulus for humans is limited. This results in a reduced number of encoded targets.

Methods

To circumvent these limitations, we propose a motion-coupled, steady-state motion visual evoked potential (SSMVEP) method. We designed a stimulus paradigm that couples both sinusoidal and square wave motions. The paradigm performs a spiral motion with a higher frequency in the form of sinusoidal wave, and alters the size of the lower frequency via the square wave form.

Results

The motion-coupled SSMVEP method could simultaneously induce stable motion frequency and coupling frequency, and there was no loss of frequency component.

Conclusions

The proposed method has been evaluated to have substantial potential for increasing the number of coding targets, which is an effective supplement to the existing studies.

A novel hybrid BCI speller based on RSVP and SSVEP paradigm

BACKGROUND AND OBJECTIVE

Steady-state visual evoked potential (SSVEP) and rapid serial visual presentation (RSVP) are useful methods in the brain-computer interface (BCI) systems. Hybrid BCI systems that combine these two approaches can enhance the proficiency of the P300 spellers.

METHODS

In this study, a new hybrid RSVP/SSVEP BCI is proposed to increase the classification accuracy and information transfer rate (ITR) as compared with the other RSVP speller paradigms. In this paradigm, RSVP (eliciting a P300 response) and SSVEP stimulations are presented in such a way that the target group of characters is identified by RSVP stimuli, and the target character is recognized by SSVEP stimuli.

RESULTS

The proposed paradigm achieved accuracy of 93.06%, and ITR of 23.41 bit/min averaged across six subjects.

CONCLUSIONS

The new hybrid system demonstrates that by using SSVEP stimulation in Triple RSVP speller paradigm, we could enhance the performance of the system as compared with the traditional Triple RSVP paradigm. Our work is the first hybrid paradigm in RSVP spellers that could obtain the higher classification accuracy and information transfer rate in comparison with the previous RSVP spellers.

Decision-Making Selector (DMS) for Integrating CCA-Based Methods to Improve Performance of SSVEP-Based BCIs

OBJECTIVE

Recent research has demonstrated improved performance of a brain-computer interface (BCI) using fusion based approaches. This paper proposes a novel decision-making selector (DMS) to integrate classification decisions of different frequency recognition methods based on canonical correlation analysis (CCA) which were used in decoding steady state visual evoked potentials (SSVEPs).

METHODS

The DMS method selects a decision more likely to be correct from two methods namely as M1 and M2 by separating the M1-false and M2-false trials. To measure the uncertainty of each decision, feature vectors were extracted using the largest and second largest correlation coefficients corresponding to all the stimulus frequencies. The proposed method was evaluated by integrating all pairs of 7 CCA-based algorithms, including CCA, individual template-based CCA (ITCCA), multi-set CCA (MsetCCA), L1-regularized multi-way CCA (L1-MCCA), filter bank CCA (FBCCA), extended CCA (ECCA), and task-related component analysis (TRCA).

MAIN RESULTS

The experimental results obtained from a 40-target dataset of thirty-five subjects showed that the proposed DMS method was validated to obtain an enhanced performance by integrating the algorithms with close accuracies.

CONCLUSION

The results suggest that the proposed DMS method is effective in integrating decisions of different methods to improve the performance of SSVEP-based BCIs.

A Hybrid Speller Design Using Eye Tracking and SSVEP Brain-Computer Interface

Steady-state visual evoked potentials (SSVEPs) have been extensively utilized to develop brain-computer interfaces (BCIs) due to the advantages of robustness, large number of commands, high classification accuracies, and information transfer rates (ITRs). However, the use of several simultaneous flickering stimuli often causes high levels of user discomfort, tiredness, annoyingness, and fatigue. Here we propose to design a stimuli-responsive hybrid speller by using electroencephalography (EEG) and video-based eye-tracking to increase user comfortability levels when presented with large numbers of simultaneously flickering stimuli. Interestingly, a canonical correlation analysis (CCA)-based framework was useful to identify target frequency with a 1 s duration of flickering signal. Our proposed BCI-speller uses only six frequencies to classify forty-eight targets, thus achieve greatly increased ITR, whereas basic SSVEP BCI-spellers use an equal number of frequencies to the number of targets. Using this speller, we obtained an average classification accuracy of 90.35 ± 3.597% with an average ITR of 184.06 ± 12.761 bits per minute in a cued-spelling task and an ITR of 190.73 ± 17.849 bits per minute in a free-spelling task. Consequently, our proposed speller is superior to the other spellers in terms of targets classified, classification accuracy, and ITR, while producing less fatigue, annoyingness, tiredness and discomfort. Together, our proposed hybrid eye tracking and SSVEP BCI-based system will ultimately enable a truly high-speed communication channel.

Optimizing the Performance of the Visual P300-Speller Through Active Mental Tasks Based on Color Distinction and Modulation of Task Difficulty

Objective: P300-speller is the most commonly used brain-computer interface (BCI) for providing a means of communication to patients with amyotrophic lateral sclerosis. However, the performance of the P300-speller BCI is still inadequate. We investigated whether the performance of P300-speller can be further improved by increasing the mental effort required of the user.

Methods: We designed two active mental tasks for a P300-speller based on a differently colored smiling cartoon-face paradigm. The tasks were based on color distinction, and their difficulty was modulated. One of the active mental tasks (DC task) required participants to focus on and distinguish the color of a target, while the other task (CN + DC task) required participants to simultaneously count the number of times a target flashed and distinguish its color.

Results: The amplitudes of the event-related potentials (ERPs) in both DC and CN + DC tasks were higher than that in the CN task. The significant difference in the amplitudes between the DC and CN tasks was observed around the parietal-central area from 440 to 800 ms (late positive component, LPC), and that between the CN + DC and CN tasks was observed around the left-frontal and right-frontal areas from 320 to 480 ms (P3a) and the parietal-central area from 480 to 800 ms (P3b and LPC). The latency of the P300 potential in the CN + DC task was significantly longer than that in the CN task at F3, Fz, F4, C4, Pz, and P4 (P < 0.05). Offline (P < 0.05 at superposing once, twice, and thrice) and online (P < 0.001) classification results showed that the average accuracies in the CN + DC task were significantly greater than that in the CN task. Similar results were found for online information transfer rates (ITRs; P < 0.001). In addition, we found that the average online accuracies in the DC task were greater than those in the CN task, although the difference was not statistically significant (P = 0.051).

Significance: The active mental task based on task difficulty modulation can significantly improve the performance of the P300-speller, and that based on color distinction shows a trend of improved performance.

Recent Advances in Hybrid Brain-Computer Interface Systems: A Technological and Quantitative Review

Brain-Computer Interface (BCI) is a system that enables users to transmit commands to the computer using their brain activity recorded by electroencephalography. In a Hybrid Brain-Computer Interface (HBCI), a BCI control signal combines with one or more BCI control signals or with Human-Machine Interface (HMI) biosignals to increase classification accuracy, boost system speed, and improve user’s satisfaction. HBCI systems are categorized according to the type of combined signals and the combination technique (simultaneous or sequential). They have been used in several applications such as cursor control, target selection, and spellers. Increasing the number of articles published in this field indicates the significance of these systems. In this paper, different HBCI combinations, their important features, and potential applications are discussed. In most cases, the combination of a BCI control signal with a HMI biosignal yields higher information transfer rate than two BCI control signals.

Towards optimal visual presentation design for hybrid EEG-fTCD brain-computer interfaces

OBJECTIVE

In this paper, we introduce a novel hybrid brain-computer interface (BCI) system that measures electrical brain activity as well as cerebral blood velocity using electroencephalography (EEG) and functional transcranial Doppler ultrasound (fTCD) respectively in response to flickering mental rotation (MR) and flickering word generation (WG) cognitive tasks as well as a fixation cross that represents the baseline. This work extends our previous approach, in which we showed that motor imagery induces simultaneous changes in EEG and fTCD to enable task discrimination; and hence, provides a design approach for a hybrid BCI. Here, we show that instead of using motor imagery, the proposed visual stimulation technique enables the design of an EEG-fTCD based BCI with higher accuracy.

APPROACH

Features based on the power spectrum of EEG and fTCD signals were calculated. Mutual information and support vector machines were used for feature selection and classification purposes.

MAIN RESULTS

EEG-fTCD combination outperformed EEG by 4.05% accuracy for MR versus baseline problem and by 5.81% accuracy for WG versus baseline problem. An average accuracy of 92.38% was achieved for MR versus WG problem using the hybrid combination. Average transmission rates of 4.39, 3.92, and 5.60 bits min^-1 were obtained for MR versus baseline, WG versus baseline, and MR versus WG problems respectively.

SIGNIFICANCE

In terms of accuracy, the current visual presentation outperforms the motor imagery visual presentation we designed before for the EEG-fTCD system by 10% accuracy for task versus task problem. Moreover, the proposed system outperforms the state of the art hybrid EEG-fNIRS BCIs in terms of accuracy and/or information transfer rate. Even though there are still limitations of the proposed system, such promising results show that the proposed hybrid system is a feasible candidate for real-time BCIs.

Complex sparse spatial filter for decoding mixed frequency and phase coded steady-state visually evoked potentials

BACKGROUND

Mixed frequency and phase coding (FPC) can achieve the significant increase of the number of commands in steady-state visual evoked potential-based brain-computer interface (SSVEP-BCI). However, the inconsistent phases of the SSVEP over channels in a trial and the existence of non-contributing channels due to noise effects can decrease accurate detection of stimulus frequency.

NEW METHOD

We propose a novel command detection method based on a complex sparse spatial filter (CSSF) by solving ℓ₁- and ℓ_2,1-regularization problems for a mixed-coded SSVEP-BCI. In particular, ℓ_2,1-regularization (aka group sparsification) can lead to the rejection of electrodes that are not contributing to the SSVEP detection.

RESULTS

A calibration data based canonical correlation analysis (CCA) and CSSF with ℓ₁- and ℓ_2,1-regularization cases were demonstrated for a 16-target stimuli with eleven subjects. The results of statistical test suggest that the proposed method with ℓ₁- and ℓ_2,1-regularization significantly achieved the highest ITR.

COMPARISON WITH EXISTING METHODS

The proposed approaches do not need any reference signals, automatically select prominent channels, and reduce the computational cost compared to the other mixed frequency-phase coding (FPC)-based BCIs.

CONCLUSIONS

The experimental results suggested that the proposed method can be usable implementing BCI effectively with reduce visual fatigue.

Development of an electrooculogram-based human-computer interface using involuntary eye movement by spatially rotating sound for communication of locked-in patients

Individuals who have lost normal pathways for communication need augmentative and alternative communication (AAC) devices. In this study, we propose a new electrooculogram (EOG)-based human-computer interface (HCI) paradigm for AAC that does not require a user’s voluntary eye movement for binary yes/no communication by patients in locked-in state (LIS). The proposed HCI uses a horizontal EOG elicited by involuntary auditory oculogyric reflex, in response to a rotating sound source. In the proposed HCI paradigm, a user was asked to selectively attend to one of two sound sources rotating in directions opposite to each other, based on the user’s intention. The user’s intentions could then be recognised by quantifying EOGs. To validate its performance, a series of experiments was conducted with ten healthy subjects, and two patients with amyotrophic lateral sclerosis (ALS). The online experimental results exhibited high-classification accuracies of 94% in both healthy subjects and ALS patients in cases where decisions were made every six seconds. The ALS patients also participated in a practical yes/no communication experiment with 26 or 30 questions with known answers. The accuracy of the experiments with questionnaires was 94%, demonstrating that our paradigm could constitute an auxiliary AAC system for some LIS patients.

An approach for brain-controlled prostheses based on Scene Graph Steady-State Visual Evoked Potentials

Brain control technology can restore communication between the brain and a prosthesis, and choosing a Brain-Computer Interface (BCI) paradigm to evoke electroencephalogram (EEG) signals is an essential step for developing this technology. In this paper, the Scene Graph paradigm used for controlling prostheses was proposed; this paradigm is based on Steady-State Visual Evoked Potentials (SSVEPs) regarding the Scene Graph of a subject's intention. A mathematic model was built to predict SSVEPs evoked by the proposed paradigm and a sinusoidal stimulation method was used to present the Scene Graph stimulus to elicit SSVEPs from subjects. Then, a 2-degree of freedom (2-DOF) brain-controlled prosthesis system was constructed to validate the performance of the Scene Graph-SSVEP (SG-SSVEP)-based BCI. The classification of SG-SSVEPs was detected via the Canonical Correlation Analysis (CCA) approach. To assess the efficiency of proposed BCI system, the performances of traditional SSVEP-BCI system were compared. Experimental results from six subjects suggested that the proposed system effectively enhanced the SSVEP responses, decreased the degradation of SSVEP strength and reduced the visual fatigue in comparison with the traditional SSVEP-BCI system. The average signal to noise ratio (SNR) of SG-SSVEP was 6.31 ± 2.64 dB, versus 3.38 ± 0.78 dB of traditional-SSVEP. In addition, the proposed system achieved good performances in prosthesis control. The average accuracy was 94.58% ± 7.05%, and the corresponding high information transfer rate (IRT) was 19.55 ± 3.07 bit/min. The experimental results revealed that the SG-SSVEP based BCI system achieves the good performance and improved the stability relative to the conventional approach.

An EOG-Based Human-Machine Interface to Control a Smart Home Environment for Patients With Severe Spinal Cord Injuries

OBJECTIVE

This paper presents an asyn-chronous electrooculography (EOG)-based human-machine interface (HMI) for smart home environmental control with the purpose of providing daily assistance for severe spinal cord injury (SCI) patients.

METHODS

The proposed HMI allows users to interact with a smart home environment through eye blinking. Specifically, several buttons, each corresponding to a control command, randomly flash on a graphical user interface. Each flash of the buttons functions as a visual cue for the user to blink. To issue a control command, the user can blink synchronously with the flashes of the corresponding button. Through detecting blinks based on the recorded EOG signal, the target button and its corresponding control command are determined. Seven SCI patients participated in an online experiment, during which the patients were required to control a smart home environment including household electrical appliances, an intelligent wheelchair, as well as a nursing bed via the proposed HMI.

RESULTS

The average false operation ratio in the control state was 4.1%, whereas during the idle state, no false operations occurred.

CONCLUSION

All SCI patients were able to control the smart home environment using the proposed EOG-based HMI with satisfactory performance in terms of the false operation ratio in both the control and the idle states.

SIGNIFICANCE

The proposed HMI offers a simple and effective approach for patients with severe SCIs to control a smart home environment. Therefore, it is promising to assist severe SCI patients in their daily lives.

Sinc-Windowing and Multiple Correlation Coefficients Improve SSVEP Recognition Based on Canonical Correlation Analysis

Canonical Correlation Analysis (CCA) is an increasingly used approach in the field of Steady-State Visually Evoked Potential (SSVEP) recognition. The efficacy of the method has been widely proven, and several variations have been proposed. However, most CCA variations tend to complicate the method, usually requiring additional user training or increasing computational load. Taking simple procedures and low computational costs may be, however, a relevant aspect, especially in view of low-cost and high-portability devices. In addition, it would be desirable that the proposed variations are as general and modular as possible to facilitate the translation of results to different algorithms and setups. In this work, we evaluated the impact of two simple, modular variations of the classical CCA method. The variations involved (i) the number of canonical correlations used for classification and (ii) the inclusion of a prefiltering step by means of sinc-windowing. We tested ten volunteers in a 4-class SSVEP setup. Both variations significantly improved classification accuracy when they were used separately or in conjunction and led to accuracy increments up to 7-8% on average and peak of 25–30%. Additionally, variations had no (variation (i)) or minimal (variation (ii)) impact on the number of algorithm steps required for each classification. Given the modular nature of the proposed variations and their positive impact on classification accuracy, they might be easily included in the design of CCA-based algorithms that are even different from ours.

Control of a 7-DOF Robotic Arm System With an SSVEP-Based BCI

Although robot technology has been successfully used to empower people who suffer from motor disabilities to increase their interaction with their physical environment, it remains a challenge for individuals with severe motor impairment, who do not have the motor control ability to move robots or prosthetic devices by manual control. In this study, to mitigate this issue, a noninvasive brain-computer interface (BCI)-based robotic arm control system using gaze based steady-state visual evoked potential (SSVEP) was designed and implemented using a portable wireless electroencephalogram (EEG) system. A 15-target SSVEP-based BCI using a filter bank canonical correlation analysis (FBCCA) method allowed users to directly control the robotic arm without system calibration. The online results from 12 healthy subjects indicated that a command for the proposed brain-controlled robot system could be selected from 15 possible choices in 4[Formula: see text]s (i.e. 2[Formula: see text]s for visual stimulation and 2[Formula: see text]s for gaze shifting) with an average accuracy of 92.78%, resulting in a 15 commands/min transfer rate. Furthermore, all subjects (even naive users) were able to successfully complete the entire move-grasp-lift task without user training. These results demonstrated an SSVEP-based BCI could provide accurate and efficient high-level control of a robotic arm, showing the feasibility of a BCI-based robotic arm control system for hand-assistance.

A Ternary Hybrid EEG-NIRS Brain-Computer Interface for the Classification of Brain Activation Patterns during Mental Arithmetic, Motor Imagery, and Idle State

The performance of a brain-computer interface (BCI) can be enhanced by simultaneously using two or more modalities to record brain activity, which is generally referred to as a hybrid BCI. To date, many BCI researchers have tried to implement a hybrid BCI system by combining electroencephalography (EEG) and functional near-infrared spectroscopy (NIRS) to improve the overall accuracy of binary classification. However, since hybrid EEG-NIRS BCI, which will be denoted by hBCI in this paper, has not been applied to ternary classification problems, paradigms and classification strategies appropriate for ternary classification using hBCI are not well investigated. Here we propose the use of an hBCI for the classification of three brain activation patterns elicited by mental arithmetic, motor imagery, and idle state, with the aim to elevate the information transfer rate (ITR) of hBCI by increasing the number of classes while minimizing the loss of accuracy. EEG electrodes were placed over the prefrontal cortex and the central cortex, and NIRS optodes were placed only on the forehead. The ternary classification problem was decomposed into three binary classification problems using the "one-versus-one" (OVO) classification strategy to apply the filter-bank common spatial patterns filter to EEG data. A 10 × 10-fold cross validation was performed using shrinkage linear discriminant analysis (sLDA) to evaluate the average classification accuracies for EEG-BCI, NIRS-BCI, and hBCI when the meta-classification method was adopted to enhance classification accuracy. The ternary classification accuracies for EEG-BCI, NIRS-BCI, and hBCI were 76.1 ± 12.8, 64.1 ± 9.7, and 82.2 ± 10.2%, respectively. The classification accuracy of the proposed hBCI was thus significantly higher than those of the other BCIs (p < 0.005). The average ITR for the proposed hBCI was calculated to be 4.70 ± 1.92 bits/minute, which was 34.3% higher than that reported for a previous binary hBCI study.

Study on the effects of brightness contrast on steady-state motion visual evoked potential

Traditional steady - state visual evoked potential (SSVEP) using light flicker stimulation may easy cause visual fatigue with a consequent reduction of recognition accuracy. In the previous study, we proposed a steady - state motion visual evoked potential (SSMVEP) brain - computer interface (BCI) method. A black - white ring - shaped checkerboard was used as the visual stimulus paradigm in this study. The movement patterns of the checkerboard included contraction and expansion. Based on the signal-to-noise ratio (SNR), recognition accuracy and anti-fatigue properties, the effects of black-white brightness contrast on the brain response were investigated. Experimental results indicated that when the contrast ratio was the highest, it could obtain higher recognition accuracy and SNR, yet it was easy to cause visual fatigue. When the contrast was the lowest, the sensitivity of the eyes against flicker reduced to the lowest, and it could reduce visual fatigue. However the recognition accuracy and SNR were low. In contrast, the appropriate contrast could guarantee high SNR and recognition accuracy, and as much as possible to reduce the visual fatigue.

Classification of Movement and Inhibition Using a Hybrid BCI

Brain-computer interfaces (BCIs) are an emerging technology that are capable of turning brain electrical activity into commands for an external device. Motor imagery (MI)—when a person imagines a motion without executing it—is widely employed in BCI devices for motor control because of the endogenous origin of its neural control mechanisms, and the similarity in brain activation to actual movements. Challenges with translating a MI-BCI into a practical device used outside laboratories include the extensive training required, often due to poor user engagement and visual feedback response delays; poor user flexibility/freedom to time the execution/inhibition of their movements, and to control the movement type (right arm vs. left leg) and characteristics (reaching vs. grabbing); and high false positive rates of motion control. Solutions to improve sensorimotor activation and user performance of MI-BCIs have been explored. Virtual reality (VR) motor-execution tasks have replaced simpler visual feedback (smiling faces, arrows) and have solved this problem to an extent. Hybrid BCIs (hBCIs) implementing an additional control signal to MI have improved user control capabilities to a limited extent. These hBCIs either fail to allow the patients to gain asynchronous control of their movements, or have a high false positive rate. We propose an immersive VR environment which provides visual feedback that is both engaging and immediate, but also uniquely engages a different cognitive process in the patient that generates event-related potentials (ERPs). These ERPs provide a key executive function for the users to execute/inhibit movements. Additionally, we propose signal processing strategies and machine learning algorithms to move BCIs toward developing long-term signal stability in patients with distinctive brain signals and capabilities to control motor signals. The hBCI itself and the VR environment we propose would help to move BCI technology outside laboratory environments for motor rehabilitation in hospitals, and potentially for controlling a prosthetic.

Development of Single-Channel Hybrid BCI System Using Motor Imagery and SSVEP

Numerous EEG-based brain-computer interface (BCI) systems that are being developed focus on novel feature extraction algorithms, classification methods and combining existing approaches to create hybrid BCIs. Several recent studies demonstrated various advantages of hybrid BCI systems in terms of an improved accuracy or number of commands available for the user. But still, BCI systems are far from realization for daily use. Having high performance with less number of channels is one of the challenging issues that persists, especially with hybrid BCI systems, where multiple channels are necessary to record information from two or more EEG signal components. Therefore, this work proposes a single-channel (C3 or C4) hybrid BCI system that combines motor imagery (MI) and steady-state visually evoked potential (SSVEP) approaches. This study demonstrates that besides MI features, SSVEP features can also be captured from C3 or C4 channel. The results show that due to rich feature information (MI and SSVEP) at these channels, the proposed hybrid BCI system outperforms both MI- and SSVEP-based systems having an average classification accuracy of 85.6 ± 7.7% in a two-class task.

Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review

In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.