A study on dynamic model of steady-state visual evoked potentials

A fused multi-subfrequency bands and CBAM SSVEP-BCI classification method based on convolutional neural network

For the brain-computer interface (BCI) system based on steady-state visual evoked potential (SSVEP), it is difficult to obtain satisfactory classification performance for short-time window SSVEP signals by traditional methods. In this paper, a fused multi-subfrequency bands and convolutional block attention module (CBAM) classification method based on convolutional neural network (CBAM-CNN) is proposed for discerning SSVEP-BCI tasks. This method extracts multi-subfrequency bands SSVEP signals as the initial input of the network model, and then carries out feature fusion on all feature inputs. In addition, CBAM is embedded in both parts of the initial input and feature fusion for adaptive feature refinement. To verify the effectiveness of the proposed method, this study uses the datasets of Inner Mongolia University of Technology (IMUT) and Tsinghua University (THU) to evaluate the performance of the proposed method. The experimental results show that the highest accuracy of CBAM-CNN reaches 0.9813 percentage point (pp). Within 0.1-2 s time window, the accuracy of CBAM-CNN is 0.0201-0.5388 (pp) higher than that of CNN, CCA-CWT-SVM, CCA-SVM, CCA-GNB, FBCCA, and CCA. Especially in the short-time window range of 0.1-1 s, the performance advantage of CBAM-CNN is more significant. The maximum information transmission rate (ITR) of CBAM-CNN is 503.87 bit/min, which is 227.53 bit/min-503.41 bit/min higher than the above six EEG decoding methods. The study further results show that CBAM-CNN has potential application value in SSVEP decoding.

Assembling global and local spatial-temporal filters to extract discriminant information of EEG in RSVP task

Objective.Brain-computer interface (BCI) system has developed rapidly in the past decade. And rapid serial visual presentation (RSVP) is an important BCI paradigm to detect the targets in high-speed image streams. For decoding electroencephalography (EEG) in RSVP task, the ensemble-model methods have better performance than the single-model ones.Approach.This study proposed a method based on ensemble learning to extract discriminant information of EEG. An extreme gradient boosting framework was utilized to sequentially generate the sub models, including one global spatial-temporal filter and a group of local ones. EEG was reshaped into a three-dimensional form by remapping the electrode dimension into a 2D array to learn the spatial-temporal features from real local space.Main results.A benchmark RSVP EEG dataset was utilized to evaluate the performance of the proposed method, where EEG data of 63 subjects were analyzed. Compared with several state-of-the-art methods, the spatial-temporal patterns of proposed method were more consistent with P300, and the proposed method can provide significantly better classification performance.Significance.The ensemble model in this study was end-to-end optimized, which can avoid error accumulation. The sub models optimized by gradient boosting theory can extract discriminant information complementarily and non-redundantly.

A novel multiple time-frequency sequential coding strategy for hybrid brain-computer interface

Background

For brain-computer interface (BCI) communication, electroencephalography provides a preferable choice due to its high temporal resolution and portability over other neural recording techniques. However, current BCIs are unable to sufficiently use the information from time and frequency domains simultaneously. Thus, we proposed a novel hybrid time-frequency paradigm to investigate better ways of using the time and frequency information.

Method

We adopt multiple omitted stimulus potential (OSP) and steady-state motion visual evoked potential (SSMVEP) to design the hybrid paradigm. A series of pre-experiments were undertaken to study factors that would influence the feasibility of the hybrid paradigm and the interaction between multiple features. After that, a novel Multiple Time-Frequencies Sequential Coding (MTFSC) strategy was introduced and explored in experiments.

Results

Omissions with multiple short and long durations could effectively elicit time and frequency features, including the multi-OSP, ERP, and SSVEP in this hybrid paradigm. The MTFSC was feasible and efficient. The preliminary online analysis showed that the accuracy and the ITR of the nine-target stimulator over thirteen subjects were 89.04% and 36.37 bits/min.

Significance

This study first combined the SSMVEP and multi-OSP in a hybrid paradigm to produce robust and abundant time features for coding BCI. Meanwhile, the MTFSC proved feasible and showed great potential in improving performance, such as expanding the number of BCI targets by better using time information in specific stimulated frequencies. This study holds promise for designing better BCI systems with a novel coding method.

Humanoid Robot Walking in Maze Controlled by SSVEP-BCI Based on Augmented Reality Stimulus

The application study of robot control based brain-computer interface (BCI) not only helps to promote the practicality of BCI but also helps to promote the advancement of robot technology, which is of great significance. Among the many obstacles, the importability of the stimulator brings much inconvenience to the robot control task. In this study, augmented reality (AR) technology was employed as the visual stimulator of steady-state visual evoked potential (SSVEP)-BCI and the robot walking experiment in the maze was designed to testify the applicability of the AR-BCI system. The online experiment was designed to complete the robot maze walking task and the robot walking commands were sent out by BCI system, in which human intentions were decoded by Filter Bank Canonical Correlation Analysis (FBCCA) algorithm. The results showed that all the 12 subjects could complete the robot walking task in the maze, which verified the feasibility of the AR-SSVEP-NAO system. This study provided an application demonstration for the robot control base on brain–computer interface, and further provided a new method for the future portable BCI system.

Multisymbol Time Division Coding for High-Frequency Steady-State Visual Evoked Potential-Based Brain-Computer Interface

The optimization of coding stimulus is a crucial factor in the study of steady-state visual evoked potential (SSVEP)-based brain-computer interface(BCI).This study proposed an encoding approach named Multi-Symbol Time Division Coding (MSTDC). This approach is based on a protocol of maximizing the distance between neural responses, which aims to encode stimulation systems implementing any number of targets with finite stimulations of different frequencies and phases. Firstly, this study designed an SSVEP-based BCI system containing forty targets with this approach. The stimulation encoding of this system was achieved with four temporal-divided stimuli that adopt the same frequency of 30 Hz and different phases. During the online experiments of twelve subjects, this system achieved an average accuracy of 96.77 ±2.47 % and an average information transfer rate (ITR) of 119.05 ± 6.11 bits/min. This study also devised an SSVEP-based BCI system containing 72 targets and proposed a Template Splicing task-related component analysis (TRCA) algorithm that utilized the dataset of the previous system containing forty targets as the training dataset. The subjects acquired an average accuracy of 86.23 ± 7.75% and an average ITR of 95.68 ± 14.19 bits/min. It can be inferred that MSTDC can encode multiple targets with limited frequencies and phases of stimuli. Meanwhile, this protocol can be effortlessly expanded into other systems and sufficiently reduce the cost of collecting training data. This study provides a feasible technique for obtaining a comfortable SSVEP-based BCI with multiple targets while maintaining high information transfer rate.

tACS facilitates flickering driving by boosting steady-state visual evoked potentials

Objective.There has become of increasing interest in transcranial alternating current stimulation (tACS) since its inception nearly a decade ago. tACS in modulating brain state is an active area of research and has been demonstrated effective in various neuropsychological and clinical domains. In the visual domain, much effort has been dedicated to brain rhythms and rhythmic stimulation, i.e. tACS. However, less is known about the interplay between the rhythmic stimulation and visual stimulation.Approach.Here, we used steady-state visual evoked potential (SSVEP), induced by flickering driving as a widely used technique for frequency-tagging, to investigate the aftereffect of tACS in healthy human subjects. Seven blocks of 64-channel electroencephalogram were recorded before and after the administration of 20min 10Hz tACS, while subjects performed several blocks of SSVEP tasks. We characterized the physiological properties of tACS aftereffect by comparing and validating the temporal, spatial, spatiotemporal and signal-to-noise ratio (SNR) patterns between and within blocks in real tACS and sham tACS.Main results.Our result revealed that tACS boosted the 10Hz SSVEP significantly. Besides, the aftereffect on SSVEP was mitigated with time and lasted up to 5 min.Significance.Our results demonstrate the feasibility of facilitating the flickering driving by external rhythmic stimulation and open a new possibility to alter the brain state in a direction by noninvasive transcranial brain stimulation.

Modulation of brain states on fractal and oscillatory power of EEG in brain-computer interfaces

Objective. Electroencephalogram (EEG) is an objective reflection of the brain activities, which provides potential possibilities for brain state estimation based on EEG characteristics. However, how to mine the effective EEG characteristics is still a distressing problem in brain state monitoring.Approach. The phase-scrambled method was used to generate images with different noise levels. Images were encoded into a rapid serial visual presentation paradigm. N-back working memory method was employed to induce and assess fatigue state. The irregular-resampling auto-spectral analysis method was adopted to extract and parameterize (exponent and offset) the characteristics of EEG fractal components, which were analyzed in the four dimensions: fatigue, sustained attention, visual noise and experimental tasks.Main results. The degree of fatigue and visual noise level had positive effects on exponent and offset in the prefrontal lobe, and the ability of sustained attention negatively affected exponent and offset. Compared with visual stimuli task, rest task induced even larger values of exponent and offset and statistically significant in the most cerebral cortex. In addition, the steady-state visual evoked potential amplitudes were negatively and positively affected by the degree of fatigue and noise levels, respectively.Significance. The conclusions of this study provide insights into the relationship between brain states and EEG characteristics. In addition, this study has the potential to provide objective methods for brain states monitoring and EEG modeling.

A Cross-Session Dataset for Collaborative Brain-Computer Interfaces Based on Rapid Serial Visual Presentation

Brain-computer interfaces (BCIs) based on rapid serial visual presentation (RSVP) have been widely used to categorize target and non-target images. However, it is still a challenge to detect single-trial event related potentials (ERPs) from electroencephalography (EEG) signals. Besides, the variability of EEG signal over time may cause difficulties of calibration in long-term system use. Recently, collaborative BCIs have been proposed to improve the overall BCI performance by fusing brain activities acquired from multiple subjects. For both individual and collaborative BCIs, feature extraction and classification algorithms that can be transferred across sessions can significantly facilitate system calibration. Although open datasets are highly efficient for developing algorithms, currently there is still a lack of datasets for a collaborative RSVP-based BCI. This paper presents a cross-session EEG dataset of a collaborative RSVP-based BCI system from 14 subjects, who were divided into seven groups. In collaborative BCI experiments, two subjects did the same target image detection tasks synchronously. All subjects participated in the same experiment twice with an average interval of ∼23 days. The results in data evaluation indicate that adequate signal processing algorithms can greatly enhance the cross-session BCI performance in both individual and collaborative conditions. Besides, compared with individual BCIs, the collaborative methods that fuse information from multiple subjects obtain significantly improved BCI performance. This dataset can be used for developing more efficient algorithms to enhance performance and practicality of a collaborative RSVP-based BCI system.

A Benchmark Dataset for RSVP-Based Brain-Computer Interfaces

This paper reports on a benchmark dataset acquired with a brain–computer interface (BCI) system based on the rapid serial visual presentation (RSVP) paradigm. The dataset consists of 64-channel electroencephalogram (EEG) data from 64 healthy subjects (sub1,…, sub64) while they performed a target image detection task. For each subject, the data contained two groups (“A” and “B”). Each group contained two blocks, and each block included 40 trials that corresponded to 40 stimulus sequences. Each sequence contained 100 images presented at 10 Hz (10 images per second). The stimulus images were street-view images of two categories: target images with human and non-target images without human. Target images were presented randomly in the stimulus sequence with a probability of 1∼4%. During the stimulus presentation, subjects were asked to search for the target images and ignore the non-target images in a subjective manner. To keep all original information, the dataset was the raw continuous data without any processing. On one hand, the dataset can be used as a benchmark dataset to compare the algorithms for target identification in RSVP-based BCIs. On the other hand, the dataset can be used to design new system diagrams and evaluate their BCI performance without collecting any new data through offline simulation. Furthermore, the dataset also provides high-quality data for characterizing and modeling event-related potentials (ERPs) and steady-state visual evoked potentials (SSVEPs) in RSVP-based BCIs. The dataset is freely available from http://bci.med.tsinghua.edu.cn/download.html.

The effect of fatigue on brain connectivity networks

In the fatigue state, the neural response characteristics of the brain might be different from those in the normal state. Brain functional connectivity analysis is an effective tool for distinguishing between different brain states. For example, comparative studies on the brain functional connectivity have the potential to reveal the functional differences in different mental states. The purpose of this study was to explore the relationship between human mental states and brain control abilities by analyzing the effect of fatigue on the brain response connectivity. In particular, the phase‐scrambling method was used to generate images with two noise levels, while the N‐back working memory task was used to induce the fatigue state in subjects. The paradigm of rapid serial visual presentation (RSVP) was used to present visual stimuli. The analysis of brain connections in the normal and fatigue states was conducted using the open‐source eConnectome toolbox. The results demonstrated that the control areas of neural responses were mainly distributed in the parietal region in both the normal and fatigue states. Compared to the normal state, the brain connectivity power in the parietal region was significantly weakened under the fatigue state, which indicates that the control ability of the brain is reduced in the fatigue state.

Concurrent electrophysiological and hemodynamic measurements of evoked neural oscillations in human visual cortex using sparsely interleaved fast fMRI and EEG

Electroencephalography (EEG) concurrently collected with functional magnetic resonance imaging (fMRI) is heavily distorted by the repetitive gradient coil switching during the fMRI acquisition. The performance of the typical template-based gradient artifact suppression method can be suboptimal because the artifact changes over time. Gradient artifact residuals also impede the subsequent suppression of ballistocardiography artifacts. Here we propose recording continuous EEG with temporally sparse fast fMRI (fast fMRI-EEG) to minimize the EEG artifacts caused by MRI gradient coil switching without significantly compromising the field-of-view and spatiotemporal resolution of fMRI. Using simultaneous multi-slice inverse imaging to achieve whole-brain fMRI with isotropic 5-mm resolution in 0.1 ​s, and performing these acquisitions once every 2 ​s, we have 95% of the duty cycle available to record EEG with substantially less gradient artifact. We found that the standard deviation of EEG signals over the entire acquisition period in fast fMRI-EEG was reduced to 54% of that in conventional concurrent echo-planar imaging (EPI) and EEG recordings (EPI-EEG) across participants. When measuring 15-Hz steady-state visual evoked potentials (SSVEPs), the baseline-normalized oscillatory neural response in fast fMRI-EEG was 2.5-fold of that in EPI-EEG. The functional MRI responses associated with the SSVEP delineated by EPI and fast fMRI were similar in the spatial distribution, the elicited waveform, and detection power. Sparsely interleaved fast fMRI-EEG provides high-quality EEG without substantially compromising the quality of fMRI in evoked response measurements, and has the potential utility for applications where the onset of the target stimulus cannot be precisely determined, such as epilepsy.

Exploration of User's Mental State Changes during Performing Brain-Computer Interface

Substantial developments have been established in the past few years for enhancing the performance of brain–computer interface (BCI) based on steady-state visual evoked potential (SSVEP). The past SSVEP-BCI studies utilized different target frequencies with flashing stimuli in many different applications. However, it is not easy to recognize user’s mental state changes when performing the SSVEP-BCI task. What we could observe was the increasing EEG power of the target frequency from the user’s visual area. BCI user’s cognitive state changes, especially in mental focus state or lost-in-thought state, will affect the BCI performance in sustained usage of SSVEP. Therefore, how to differentiate BCI users’ physiological state through exploring their neural activities changes while performing SSVEP is a key technology for enhancing the BCI performance. In this study, we designed a new BCI experiment which combined working memory task into the flashing targets of SSVEP task using 12 Hz or 30 Hz frequencies. Through exploring the EEG activity changes corresponding to the working memory and SSVEP task performance, we can recognize if the user’s cognitive state is in mental focus or lost-in-thought. Experiment results show that the delta (1–4 Hz), theta (4–7 Hz), and beta (13–30 Hz) EEG activities increased more in mental focus than in lost-in-thought state at the frontal lobe. In addition, the powers of the delta (1–4 Hz), alpha (8–12 Hz), and beta (13–30 Hz) bands increased more in mental focus in comparison with the lost-in-thought state at the occipital lobe. In addition, the average classification performance across subjects for the KNN and the Bayesian network classifiers were observed as 77% to 80%. These results show how mental state changes affect the performance of BCI users. In this work, we developed a new scenario to recognize the user’s cognitive state during performing BCI tasks. These findings can be used as the novel neural markers in future BCI developments.

A Method for Tracking the Time Evolution of Steady-State Evoked Potentials

Neural entrainment refers to the synchronization of neural activity to the periodicity of sensory stimuli. This synchronization defines the generation of steady-state evoked responses (i.e., oscillations in the electroencephalogram phase-locked to the driving stimuli). The classic interpretation of the amplitude of the steady-state evoked responses assumes a stereotypical time-invariant neural response plus random background fluctuations, such that averaging over repeated presentations of the stimulus recovers the stereotypical response. This approach ignores the dynamics of the steady-state, as in the case of the adaptation elicited by prolonged exposures to the stimulus. To analyze the dynamics of steady-state responses, it can be assumed that the time evolution of the response amplitude is the same in different stimulation runs separated by sufficiently long breaks. Based on this assumption, a method to characterize the time evolution of steady-state responses is presented. A sufficiently large number of recordings are acquired in response to the same experimental condition. Experimental runs (recordings) are column-wise averaged (i.e., runs are averaged but epoch within recordings are not averaged with the preceding segments). The column-wise averaging allows analysis of steady-state responses in recordings with remarkably high signal-to-noise ratios. Therefore, the averaged signal provides an accurate representation of the time evolution of the steady-state response, which can be analyzed in both the time and frequency domains. In this study, a detailed description of the method is provided, using steady-state visually evoked potentials as an example of a response. Advantages and caveats are evaluated based on a comparison with single-trial methods designed to analyze neural entrainment.