Two brains, one target: Design of a multi-level information fusion model based on dual-subject RSVP

BACKGROUND

Rapid serial visual presentation (RSVP) based brain-computer interface (BCI) is widely used to categorize the target and non-target images. The available information limits the prediction accuracy of single-trial using single-subject electroencephalography (EEG) signals. New Method. Hyperscanning is a new manner to record two or more subjects' signals simultaneously. So we designed a multi-level information fusion model for target image detection based on dual-subject RSVP, namely HyperscanNet. The two modules of this model fuse the data and features of the two subjects at the data and feature layers. A chunked long and short-term memory artificial neural network (LSTM) was used in the time dimension to extract features at different periods separately, completing fine-grained underlying feature extraction. While the feature layer is fused, some plain operations are used to complete the fusion of the data layer to ensure that important information is not missed.

RESULTS

Experimental results show that the F1-score (the harmonic mean of precision and recall) of this method with best group of channels and segment length is 82.76%. Comparison with existing methods. This method improves the F1-score by at least 5% compared to single-subject target detection.

CONCLUSIONS

Target detection can be accomplished by the two subjects' collaboration to achieve a higher and more stable F1-score than a single subject.

Feature Selection Combining Filter and Wrapper Methods for Motor-Imagery Based Brain-Computer Interfaces

Motor imagery (MI) based brain-computer interfaces help patients with movement disorders to regain the ability to control external devices. Common spatial pattern (CSP) is a popular algorithm for feature extraction in decoding MI tasks. However, due to noise and nonstationarity in electroencephalography (EEG), it is not optimal to combine the corresponding features obtained from the traditional CSP algorithm. In this paper, we designed a novel CSP feature selection framework that combines the filter method and the wrapper method. We first evaluated the importance of every CSP feature by the infinite latent feature selection method. Meanwhile, we calculated Wasserstein distance between feature distributions of the same feature under different tasks. Then, we redefined the importance of every CSP feature based on two indicators mentioned above, which eliminates half of CSP features to create a new CSP feature subspace according to the new importance indicator. At last, we designed the improved binary gravitational search algorithm (IBGSA) by rebuilding its transfer function and applied IBGSA on the new CSP feature subspace to find the optimal feature set. To validate the proposed method, we conducted experiments on three public BCI datasets and performed a numerical analysis of the proposed algorithm for MI classification. The accuracies were comparable to those reported in related studies and the presented model outperformed other methods in literature on the same underlying data.

Optimization of Model Training Based on Iterative Minimum Covariance Determinant In Motor-Imagery BCI

The common spatial patterns (CSP) algorithm is one of the most frequently used and effective spatial filtering methods for extracting relevant features for use in motor imagery brain-computer interfaces (MI-BCIs). However, the inherent defect of the traditional CSP algorithm is that it is highly sensitive to potential outliers, which adversely affects its performance in practical applications. In this work, we propose a novel feature optimization and outlier detection method for the CSP algorithm. Specifically, we use the minimum covariance determinant (MCD) to detect and remove outliers in the dataset, then we use the Fisher score to evaluate and select features. In addition, in order to prevent the emergence of new outliers, we propose an iterative minimum covariance determinant (IMCD) algorithm. We evaluate our proposed algorithm in terms of iteration times, classification accuracy and feature distribution using two BCI competition datasets. The experimental results show that the average classification performance of our proposed method is 12% and 22.9% higher than that of the traditional CSP method in two datasets ([Formula: see text]), and our proposed method obtains better performance in comparison with other competing methods. The results show that our method improves the performance of MI-BCI systems.

The Influence of Visual Attention on The Performance of A Novel Tactile P300 Brain-Computer Interface with Cheeks-Stim Paradigm

Tactile P300 brain-computer interface (BCI) generally has a worse accuracy and information transfer rate (ITR) than the visual-based BCI. It may be due to the fact that human beings have a relatively poor tactile perception. This study investigated the influence of visual attention on the performance of a tactile P300 BCI. We designed our paradigms based on a novel cheeks-stim paradigm which attached the stimulators on the subject's cheeks. Two paradigms were designed as follows: a paradigm with no visual attention and another paradigm with visual attention to the target position. Eleven subjects were invited to perform the two paradigms. We also recorded and analyzed the eyeball movement data during the paradigm with visual attention to explore whether the eyeball movement would have an effect on the BCI classification. The average online accuracy was 89.09% for the paradigm with visual attention, which was significantly higher than that of the paradigm with no visual attention (70.45%). Significant difference in ITR was also found between the two paradigms ([Formula: see text]). The results demonstrated that visual attention was an effective method to improve the performance of tactile P300 BCI. Our findings suggested that it may be feasible to complete an efficient tactile BCI system by adding visual attention.

The Effect of the Graphic Structures of Humanoid Robot on N200 and P300 Potentials

Humanoid robots are widely used in brain computer interface (BCI). Using a humanoid robot stimulus could increase the amplitude of event-related potentials (ERPs), which improves BCI performance. Since a humanoid robot contains many human elements, the element that increases the ERPs amplitude is unclear, and how to test the effect of it on the brain is a problem. This study used different graphic structures of an NAO humanoid robot to design three types of robot stimuli: a global robot, its local information, and its topological action. Ten subjects first conducted an odd-ball-based BCI (OD-BCI) by applying these stimuli. Then, they accomplished a delayed matching-to-sample task (DMST) that was used to specialize the encoding and retrieval phases of the OD-BCI task. In the retrieval phase of the DMST, the global stimulus induces the largest N200 and P300 potentials with the shortest latencies in the frontal, central, and occipital areas. This finding is in accordance with the P300 and classification performance of the OD-BCI task. When induced by the local stimulus, the subjects responded faster and more accurately in the retrieval phase of the DMST than in the other two conditions, indicating that the local stimulus improved the subject's responses. These results indicate that the OD-BCI task causes subject's retrieval work when the subject recognizes and outputs the stimulus. The global stimulus that contains topological and local elements could make brain react faster and induce larger ERPs, this finding could be used during the development of visual stimuli to improve BCI performance.

Recognition of the idle state based on a novel IFB-OCN method for an asynchronous brain-computer interface

BACKGROUND

A major difficulty for the asynchronous brain-computer interface (BCI) lies in the accurate recognition of the control and idle states. Although subject's attention level was found to be different in these states, the validity of recognizing them using attention features has not been studied.

NEW METHODS

This paper proposed a novel Individualized Frequency Band based Optimized Complex Network (IFB-OCN) method to enhance the performance of discriminating the control and idle states. The IFB-OCN method extracted the attention features from a single FPz channel, selected the first three individualized frequency bands with the highest accuracies, and integrated the features of these bands for classification.

RESULTS

The performance was evaluated using a steady-state visual evoked potential (SSVEP)-based BCI task. In the offline evaluation, the IFB-OCN method achieved the highest average accuracy of 93.5 % with the data length of 4 s, and achieved the highest information transfer rate (ITR) of 47.3 bits/min with the data length of 0.5 s. In the simulated online evaluation, the IFB-OCN method obtained a true positive rate (TPR) of 89.8 % and a true negative rate (TNR) of 86.2 %.

COMPARISON WITH EXISTING METHODS

The proposed IFB-OCN method recognized the control and idle states using a single FPz channel rather than the occipital channels, and outperformed the existing algorithms in the accuracy of detecting the attention level.

CONCLUSIONS

These results demonstrate that the proposed IFB-OCN method is efficient in recognizing the idle state and has a great potential for enhancing the asynchronous BCIs.

Separable EEG Features Induced by Timing Prediction for Active Brain-Computer Interfaces

Brain–computer interfaces (BCI) have witnessed a rapid development in recent years. However, the active BCI paradigm is still underdeveloped with a lack of variety. It is imperative to adapt more voluntary mental activities for the active BCI control, which can induce separable electroencephalography (EEG) features. This study aims to demonstrate the brain function of timing prediction, i.e., the expectation of upcoming time intervals, is accessible for BCIs. Eighteen subjects were selected for this study. They were trained to have a precise idea of two sub-second time intervals, i.e., 400 ms and 600 ms, and were asked to measure a time interval of either 400 ms or 600 ms in mind after a cue onset. The EEG features induced by timing prediction were analyzed and classified using the combined discriminative canonical pattern matching and common spatial pattern. It was found that the ERPs in low-frequency (0~4 Hz) and energy in high-frequency (20~60 Hz) were separable for distinct timing predictions. The accuracy reached the highest of 93.75% with an average of 76.45% for the classification of 400 vs. 600 ms timing. This study first demonstrates that the cognitive EEG features induced by timing prediction are detectable and separable, which is feasible to be used in active BCIs controls and can broaden the category of BCIs.

A TrAdaBoost Method for Detecting Multiple Subjects' N200 and P300 Potentials Based on Cross-Validation and an Adaptive Threshold

Traditional training methods need to collect a large amount of data for every subject to train a subject-specific classifier, which causes subjects fatigue and training burden. This study proposes a novel training method, TrAdaBoost based on cross-validation and an adaptive threshold (CV-T-TAB), to reduce the amount of data required for training by selecting and combining multiple subjects' classifiers that perform well on a new subject to train a classifier. This method adopts cross-validation to extend the amount of the new subject's training data and sets an adaptive threshold to select the optimal combination of the classifiers. Twenty-five subjects participated in the N200- and P300-based brain-computer interface. The study compares CV-T-TAB to five traditional training methods by testing them on the training of a support vector machine. The accuracy, information transfer rate, area under the curve, recall and precision are used to evaluate the performances under nine conditions with different amounts of data. CV-T-TAB outperforms the other methods and retains a high accuracy even when the amount of data is reduced to one-third of the original amount. The results imply that CV-T-TAB is effective in improving the performance of a subject-specific classifier with a small amount of data by adopting multiple subjects' classifiers, which reduces the training cost.

Effect of the concreteness of robot motion visual stimulus on an event-related potential-based brain-computer interface

Event-related potential (ERP)-based brain-computer interface (BCI) has been widely used in robot control. Increasing the amplitude of the ERPs is key for improving the performance of ERP-based BCI. However, using images of robot motion as visual stimuli has not been studied widely. The aim of this study is to explore the concreteness of robot motion images on ERPs. Fifteen subjects used five kinds of visual spellers employing different images as visual stimuli: squares, arrows, a single kind of robot motion, multiple kinds of robot motions, and multiple kinds of robot motions with arrows. The three robot motion stimuli induced larger N200 and P300 potentials than non-robot motion stimuli. The topography shows that robot motion stimuli also evoke stronger negativities in the anterior and occipital areas. Concrete images provide more information to the subject about the robot motion, which might help the brain extract the meaning of the image more automatically. We use a support vector machine to detect the subject's intentions. There is substantial improvement in the classification performance when using robot motion images as visual stimuli, which implies that concrete visual stimuli improve the performance of the ERP-based BCI.

A Dynamic Window Recognition Algorithm for SSVEP-Based Brain–Computer Interfaces Using a Spatio-Temporal Equalizer

The past decade has witnessed rapid development in the field of brain–computer interfaces (BCIs). While the performance is no longer the biggest bottleneck in the BCI application, the tedious training process and the poor ease-of-use have become the most significant challenges. In this study, a spatio-temporal equalization dynamic window (STE-DW) recognition algorithm is proposed for steady-state visual evoked potential (SSVEP)-based BCIs. The algorithm can adaptively control the stimulus time while maintaining the recognition accuracy, which significantly improves the information transfer rate (ITR) and enhances the adaptability of the system to different subjects. Specifically, a spatio-temporal equalization algorithm is used to reduce the adverse effects of spatial and temporal correlation of background noise. Based on the theory of multiple hypotheses testing, a stimulus termination criterion is used to adaptively control the dynamic window. The offline analysis which used a benchmark dataset and an offline dataset collected from 16 subjects demonstrated that the STE-DW algorithm is superior to the filter bank canonical correlation analysis (FBCCA), canonical variates with autoregressive spectral analysis (CVARS), canonical correlation analysis (CCA) and CCA reducing variation (CCA-RV) algorithms in terms of accuracy and ITR. The results show that in the benchmark dataset, the STE-DW algorithm achieved an average ITR of 134 bits/min, which exceeds the FBCCA, CVARS, CCA and CCA-RV. In off-line experiments, the STE-DW algorithm also achieved an average ITR of 116 bits/min. In addition, the online experiment also showed that the STE-DW algorithm can effectively expand the number of applicable users of the SSVEP-based BCI system. We suggest that the STE-DW algorithm can be used as a reliable identification algorithm for training-free SSVEP-based BCIs, because of the good balance between ease of use, recognition accuracy, ITR and user applicability.

A Dual Stimuli Approach Combined with Convolutional Neural Network to Improve Information Transfer Rate of Event-Related Potential-Based Brain-Computer Interface

Increasing command generation rate of an event-related potential-based brain-robot system is challenging, because of limited information transfer rate of a brain-computer interface system. To improve the rate, we propose a dual stimuli approach that is flashing a robot image and is scanning another robot image simultaneously. Two kinds of event-related potentials, N200 and P300 potentials, evoked in this dual stimuli condition are decoded by a convolutional neural network. Compared with the traditional approaches, this proposed approach significantly improves the online information transfer rate from 23.0 or 17.8 to 39.1 bits/min at an accuracy of 91.7%. These results suggest that combining multiple types of stimuli to evoke distinguishable ERPs might be a promising direction to improve the command generation rate in the brain-computer interface.

Multivariate weighted recurrence network analysis of EEG signals from ERP-based smart home system

Smart home has been widely used to improve the living quality of people. Recently, the brain-computer interface (BCI) contributes greatly to the smart home system. We design a BCI-based smart home system, in which the event-related potentials (ERP) are induced by the image interface based on the oddball paradigm. Then, we investigate the influence of mental fatigue on the ERP classification by the Fisher linear discriminant analysis. The results indicate that the classification accuracy of ERP decreases as the brain evolves from the normal stage to the mental fatigue stage. In order to probe into the difference of the brain, cognitive process between mental fatigue and normal states, we construct multivariate weighted recurrence networks and analyze the variation of the weighted clustering coefficient and weighted global efficiency corresponding to these two brain states. The findings suggest that these two network metrics allow distinguishing normal and mental fatigue states and yield novel insights into the brain fatigue behavior resulting from a long use of the ERP-based smart home system. These properties render the multivariate recurrence network, particularly useful for analyzing electroencephalographic recordings from the ERP-based smart home system.

Neuronal Activation Detection Using Vector Phase Analysis with Dual Threshold Circles: A Functional Near-Infrared Spectroscopy Study

In this paper, a new vector phase diagram differentiating the initial decreasing phase (i.e. initial dip) and the delayed hemodynamic response (HR) phase of oxy-hemoglobin changes ( Δ HbO) of functional near-infrared spectroscopy (fNIRS) is developed. The vector phase diagram displays the trajectories of Δ HbO and deoxy-hemoglobin changes ( Δ HbR), as orthogonal components, in the Δ HbO- Δ HbR polar coordinates. To determine the occurrence of an initial dip, dual threshold circles (an inner circle from the resting state, an outer circle from the peak values of the initial dip and the main HR) are incorporated into the phase diagram for making decisions. The proposed scheme is then applied to a brain-computer interface scheme, and its performance is evaluated in classifying two finger tapping tasks (right-hand thumb and little finger) from the left motor cortex. Three gamma functions are used to model the initial dip, the main HR, and the undershoot in generating the designed HR function. In classifying two tapping tasks, the signal mean and signal minimum values during 0-2.5 s, as features of initial dip, are used. The linear discriminant analysis was utilized as a classifier. The experimental results show that the active brain locations of the two tasks were quite distinctive ( p < 0.05 ), and moreover, spatially specific if using the initial dip map at 4 s in comparison to the map of HRs at 14 s. Also, the average classification accuracy was improved from 59% to 74.9% when using the phase diagram of dual threshold circles.

Control of a 7-DOF Robotic Arm System With an SSVEP-Based BCI

Although robot technology has been successfully used to empower people who suffer from motor disabilities to increase their interaction with their physical environment, it remains a challenge for individuals with severe motor impairment, who do not have the motor control ability to move robots or prosthetic devices by manual control. In this study, to mitigate this issue, a noninvasive brain-computer interface (BCI)-based robotic arm control system using gaze based steady-state visual evoked potential (SSVEP) was designed and implemented using a portable wireless electroencephalogram (EEG) system. A 15-target SSVEP-based BCI using a filter bank canonical correlation analysis (FBCCA) method allowed users to directly control the robotic arm without system calibration. The online results from 12 healthy subjects indicated that a command for the proposed brain-controlled robot system could be selected from 15 possible choices in 4[Formula: see text]s (i.e. 2[Formula: see text]s for visual stimulation and 2[Formula: see text]s for gaze shifting) with an average accuracy of 92.78%, resulting in a 15 commands/min transfer rate. Furthermore, all subjects (even naive users) were able to successfully complete the entire move-grasp-lift task without user training. These results demonstrated an SSVEP-based BCI could provide accurate and efficient high-level control of a robotic arm, showing the feasibility of a BCI-based robotic arm control system for hand-assistance.

Object Extraction in Cluttered Environments via a P300-Based IFCE

One of the fundamental issues for robot navigation is to extract an object of interest from an image. The biggest challenges for extracting objects of interest are how to use a machine to model the objects in which a human is interested and extract them quickly and reliably under varying illumination conditions. This article develops a novel method for segmenting an object of interest in a cluttered environment by combining a P300-based brain computer interface (BCI) and an improved fuzzy color extractor (IFCE). The induced P300 potential identifies the corresponding region of interest and obtains the target of interest for the IFCE. The classification results not only represent the human mind but also deliver the associated seed pixel and fuzzy parameters to extract the specific objects in which the human is interested. Then, the IFCE is used to extract the corresponding objects. The results show that the IFCE delivers better performance than the BP network or the traditional FCE. The use of a P300-based IFCE provides a reliable solution for assisting a computer in identifying an object of interest within images taken under varying illumination intensities.

Progress in EEG-Based Brain Robot Interaction Systems

The most popular noninvasive Brain Robot Interaction (BRI) technology uses the electroencephalogram- (EEG-) based Brain Computer Interface (BCI), to serve as an additional communication channel, for robot control via brainwaves. This technology is promising for elderly or disabled patient assistance with daily life. The key issue of a BRI system is to identify human mental activities, by decoding brainwaves, acquired with an EEG device. Compared with other BCI applications, such as word speller, the development of these applications may be more challenging since control of robot systems via brainwaves must consider surrounding environment feedback in real-time, robot mechanical kinematics, and dynamics, as well as robot control architecture and behavior. This article reviews the major techniques needed for developing BRI systems. In this review article, we first briefly introduce the background and development of mind-controlled robot technologies. Second, we discuss the EEG-based brain signal models with respect to generating principles, evoking mechanisms, and experimental paradigms. Subsequently, we review in detail commonly used methods for decoding brain signals, namely, preprocessing, feature extraction, and feature classification, and summarize several typical application examples. Next, we describe a few BRI applications, including wheelchairs, manipulators, drones, and humanoid robots with respect to synchronous and asynchronous BCI-based techniques. Finally, we address some existing problems and challenges with future BRI techniques.