P300 Speller Performance Predictor Based on RSVP Multi-feature

Evaluation of Different Types of Stimuli in an Event-Related Potential-Based Brain-Computer Interface Speller under Rapid Serial Visual Presentation

Rapid serial visual presentation (RSVP) is currently a suitable gaze-independent paradigm for controlling visual brain-computer interfaces (BCIs) based on event-related potentials (ERPs), especially for users with limited eye movement control. However, unlike gaze-dependent paradigms, gaze-independent ones have received less attention concerning the specific choice of visual stimuli that are used. In gaze-dependent BCIs, images of faces-particularly those tinted red-have been shown to be effective stimuli. This study aims to evaluate whether the colour of faces used as visual stimuli influences ERP-BCI performance under RSVP. Fifteen participants tested four conditions that varied only in the visual stimulus used: grey letters (GL), red famous faces with letters (RFF), green famous faces with letters (GFF), and blue famous faces with letters (BFF). The results indicated significant accuracy differences only between the GL and GFF conditions, unlike prior gaze-dependent studies. Additionally, GL achieved higher comfort ratings compared with other face-related conditions. This study highlights that the choice of stimulus type impacts both performance and user comfort, suggesting implications for future ERP-BCI designs for users requiring gaze-independent systems.

Semantic image sorting method for RSVP presentation

Objective.The rapid serial visual presentation (RSVP) paradigm, which is based on the electroencephalogram (EEG) technology, is an effective approach for object detection. It aims to detect the event-related potentials (ERP) components evoked by target images for rapid identification. However, the object detection performance within this paradigm is affected by the visual disparity between adjacent images in a sequence. Currently, there is no objective metric to quantify this visual difference. Consequently, a reliable image sorting method is required to ensure the generation of a smooth sequence for effective presentation.Approach. In this paper, we propose a novel semantic image sorting method for sorting RSVP sequences, which aims at generating sequences that are perceptually smoother in terms of the human visual experience.Main results. We conducted a comparative analysis between our method and two existing methods for generating RSVP sequences using both qualitative and quantitative assessments. A qualitative evaluation revealed that the sequences generated by our method were smoother in subjective vision and were more effective in evoking stronger ERP components than those generated by the other two methods. Quantitatively, our method generated semantically smoother sequences than the other two methods. Furthermore, we employed four advanced approaches to classify single-trial EEG signals evoked by each of the three methods. The classification results of the EEG signals evoked by our method were superior to those of the other two methods.Significance. In summary, the results indicate that the proposed method can significantly enhance the object detection performance in RSVP-based sequences.

Signal alignment for cross-datasets in P300 brain-computer interfaces

Objective.Transfer learning has become an important issue in the brain-computer interface (BCI) field, and studies on subject-to-subject transfer within the same dataset have been performed. However, few studies have been performed on dataset-to-dataset transfer, including paradigm-to-paradigm transfer. In this study, we propose a signal alignment (SA) for P300 event-related potential (ERP) signals that is intuitive, simple, computationally less expensive, and can be used for cross-dataset transfer learning.Approach.We proposed a linear SA that uses the P300's latency, amplitude scale, and reverse factor to transform signals. For evaluation, four datasets were introduced (two from conventional P300 Speller BCIs, one from a P300 Speller with face stimuli, and the last from a standard auditory oddball paradigm).Results.Although the standard approach without SA had an average precision (AP) score of 25.5%, the approach demonstrated a 35.8% AP score, and we observed that the number of subjects showing improvement was 36.0% on average. Particularly, we confirmed that the Speller dataset with face stimuli was more comparable with other datasets.Significance.We proposed a simple and intuitive way to align ERP signals that uses the characteristics of ERP signals. The results demonstrated the feasibility of cross-dataset transfer learning even between datasets with different paradigms.

A causal perspective on brainwave modeling for brain-computer interfaces

Objective. Machine learning (ML) models have opened up enormous opportunities in the field of brain-computer Interfaces (BCIs). Despite their great success, they usually face severe limitations when they are employed in real-life applications outside a controlled laboratory setting.Approach. Mixing causal reasoning, identifying causal relationships between variables of interest, with brainwave modeling can change one's viewpoint on some of these major challenges which can be found in various stages in the ML pipeline, ranging from data collection and data pre-processing to training methods and techniques.Main results. In this work, we employ causal reasoning and present a framework aiming to breakdown and analyze important challenges of brainwave modeling for BCIs.Significance. Furthermore, we present how general ML practices as well as brainwave-specific techniques can be utilized and solve some of these identified challenges. And finally, we discuss appropriate evaluation schemes in order to measure these techniques' performance and efficiently compare them with other methods that will be developed in the future.

BCI-Utility Metric for Asynchronous P300 Brain-Computer Interface Systems

The Brain-Computer Interface (BCI) was envisioned as an assistive technology option for people with severe movement impairments. The traditional synchronous event-related potential (ERP) BCI design uses a fixed communication speed and is vulnerable to variations in attention. Recent ERP BCI designs have added asynchronous features, including abstention and dynamic stopping, but it remains a open question of how to evaluate asynchronous BCI performance. In this work, we build on the BCI-Utility metric to create the first evaluation metric with special consideration of the asynchronous features of self-paced BCIs. This metric considers accuracy as all of the following three - probability of a correct selection when a selection was intended, probability of making a selection when a selection was intended, and probability of an abstention when an abstention was intended. Further, it considers the average time required for a selection when using dynamic stopping and the proportion of intended selections versus abstentions. We establish the validity of the derived metric via extensive simulations, and illustrate and discuss its practical usage on real-world BCI data. We describe the relative contribution of different inputs with plots of BCI-Utility curves under different parameter settings. Generally, the BCI-Utility metric increases as any of the accuracy values increase and decreases as the expected time for an intended selection increases. Furthermore, in many situations, we find shortening the expected time of an intended selection is the most effective way to improve the BCI-Utility, which necessitates the advancement of asynchronous BCI systems capable of accurate abstention and dynamic stopping.

EEG Dataset for RSVP and P300 Speller Brain-Computer Interfaces

As attention to deep learning techniques has grown, many researchers have attempted to develop ready-to-go brain-computer interfaces (BCIs) that include automatic processing pipelines. However, to do so, a large and clear dataset is essential to increase the model's reliability and performance. Accordingly, our electroencephalogram (EEG) dataset for rapid serial visual representation (RSVP) and P300 speller may contribute to increasing such BCI research. We validated our dataset with respect to features and accuracy. For the RSVP, the participants (N = 50) achieved about 92% mean target detection accuracy. At the feature level, we observed notable ERPs (at 315 ms in the RSVP; at 262 ms in the P300 speller) during target events compared to non-target events. Regarding P300 speller performance, the participants (N = 55) achieved about 92% mean accuracy. In addition, P300 speller performance over trial repetitions up to 15 was explored. The presented dataset could potentially improve P300 speller applications. Further, it may be used to evaluate feature extraction and classification algorithm effectively, such as for cross-subjects/cross-datasets, and even for the cross-paradigm BCI model.

See, Hear, or Feel - to Speak: A Versatile Multiple-Choice Functional Near-Infrared Spectroscopy-Brain-Computer Interface Feasible With Visual, Auditory, or Tactile Instructions

Severely motor-disabled patients, such as those suffering from the so-called "locked-in" syndrome, cannot communicate naturally. They may benefit from brain-computer interfaces (BCIs) exploiting brain signals for communication and therewith circumventing the muscular system. One BCI technique that has gained attention recently is functional near-infrared spectroscopy (fNIRS). Typically, fNIRS-based BCIs allow for brain-based communication via voluntarily modulation of brain activity through mental task performance guided by visual or auditory instructions. While the development of fNIRS-BCIs has made great progress, the reliability of fNIRS-BCIs across time and environments has rarely been assessed. In the present fNIRS-BCI study, we tested six healthy participants across three consecutive days using a straightforward four-choice fNIRS-BCI communication paradigm that allows answer encoding based on instructions using various sensory modalities. To encode an answer, participants performed a motor imagery task (mental drawing) in one out of four time periods. Answer encoding was guided by either the visual, auditory, or tactile sensory modality. Two participants were tested outside the laboratory in a cafeteria. Answers were decoded from the time course of the most-informative fNIRS channel-by-chromophore combination. Across the three testing days, we obtained mean single- and multi-trial (joint analysis of four consecutive trials) accuracies of 62.5 and 85.19%, respectively. Obtained multi-trial accuracies were 86.11% for visual, 80.56% for auditory, and 88.89% for tactile sensory encoding. The two participants that used the fNIRS-BCI in a cafeteria obtained the best single- (72.22 and 77.78%) and multi-trial accuracies (100 and 94.44%). Communication was reliable over the three recording sessions with multi-trial accuracies of 86.11% on day 1, 86.11% on day 2, and 83.33% on day 3. To gauge the trade-off between number of optodes and decoding accuracy, averaging across two and three promising fNIRS channels was compared to the one-channel approach. Multi-trial accuracy increased from 85.19% (one-channel approach) to 91.67% (two-/three-channel approach). In sum, the presented fNIRS-BCI yielded robust decoding results using three alternative sensory encoding modalities. Further, fNIRS-BCI communication was stable over the course of three consecutive days, even in a natural (social) environment. Therewith, the developed fNIRS-BCI demonstrated high flexibility, reliability and robustness, crucial requirements for future clinical applicability.

P300 Brain-Computer Interface-Based Drone Control in Virtual and Augmented Reality

Since the emergence of head-mounted displays (HMDs), researchers have attempted to introduce virtual and augmented reality (VR, AR) in brain–computer interface (BCI) studies. However, there is a lack of studies that incorporate both AR and VR to compare the performance in the two environments. Therefore, it is necessary to develop a BCI application that can be used in both VR and AR to allow BCI performance to be compared in the two environments. In this study, we developed an opensource-based drone control application using P300-based BCI, which can be used in both VR and AR. Twenty healthy subjects participated in the experiment with this application. They were asked to control the drone in two environments and filled out questionnaires before and after the experiment. We found no significant (p > 0.05) difference in online performance (classification accuracy and amplitude/latency of P300 component) and user experience (satisfaction about time length, program, environment, interest, difficulty, immersion, and feeling of self-control) between VR and AR. This indicates that the P300 BCI paradigm is relatively reliable and may work well in various situations.

An Open Source-Based BCI Application for Virtual World Tour and Its Usability Evaluation

Brain-computer interfaces can provide a new communication channel and control functions to people with restricted movements. Recent studies have indicated the effectiveness of brain-computer interface (BCI) applications. Various types of applications have been introduced so far in this field, but the number of those available to the public is still insufficient. Thus, there is a need to expand the usability and accessibility of BCI applications. In this study, we introduce a BCI application for users to experience a virtual world tour. This software was built on three open-source environments and is publicly available through the GitHub repository. For a usability test, 10 healthy subjects participated in an electroencephalography (EEG) experiment and evaluated the system through a questionnaire. As a result, all the participants successfully played the BCI application with 96.6% accuracy with 20 blinks from two sessions and gave opinions on its usability (e.g., controllability, completeness, comfort, and enjoyment) through the questionnaire. We believe that this open-source BCI world tour system can be used in both research and entertainment settings and hopefully contribute to open science in the BCI field.

A Survey on Deep Learning-Based Short/Zero-Calibration Approaches for EEG-Based Brain-Computer Interfaces

Brain-computer interfaces (BCIs) utilizing machine learning techniques are an emerging technology that enables a communication pathway between a user and an external system, such as a computer. Owing to its practicality, electroencephalography (EEG) is one of the most widely used measurements for BCI. However, EEG has complex patterns and EEG-based BCIs mostly involve a cost/time-consuming calibration phase; thus, acquiring sufficient EEG data is rarely possible. Recently, deep learning (DL) has had a theoretical/practical impact on BCI research because of its use in learning representations of complex patterns inherent in EEG. Moreover, algorithmic advances in DL facilitate short/zero-calibration in BCI, thereby suppressing the data acquisition phase. Those advancements include data augmentation (DA), increasing the number of training samples without acquiring additional data, and transfer learning (TL), taking advantage of representative knowledge obtained from one dataset to address the so-called data insufficiency problem in other datasets. In this study, we review DL-based short/zero-calibration methods for BCI. Further, we elaborate methodological/algorithmic trends, highlight intriguing approaches in the literature, and discuss directions for further research. In particular, we search for generative model-based and geometric manipulation-based DA methods. Additionally, we categorize TL techniques in DL-based BCIs into explicit and implicit methods. Our systematization reveals advances in the DA and TL methods. Among the studies reviewed herein, ~45% of DA studies used generative model-based techniques, whereas ~45% of TL studies used explicit knowledge transferring strategy. Moreover, based on our literature review, we recommend an appropriate DA strategy for DL-based BCIs and discuss trends of TLs used in DL-based BCIs.

Target Detection Using Ternary Classification During a Rapid Serial Visual Presentation Task Using Magnetoencephalography Data

Background: The rapid serial visual presentation (RSVP) paradigm is a high-speed paradigm of brain–computer interface (BCI) applications. The target stimuli evoke event-related potential (ERP) activity of odd-ball effect, which can be used to detect the onsets of targets. Thus, the neural control can be produced by identifying the target stimulus. However, the ERPs in single trials vary in latency and length, which makes it difficult to accurately discriminate the targets against their neighbors, the near-non-targets. Thus, it reduces the efficiency of the BCI paradigm.

Methods: To overcome the difficulty of ERP detection against their neighbors, we proposed a simple but novel ternary classification method to train the classifiers. The new method not only distinguished the target against all other samples but also further separated the target, near-non-target, and other, far-non-target samples. To verify the efficiency of the new method, we performed the RSVP experiment. The natural scene pictures with or without pedestrians were used; the ones with pedestrians were used as targets. Magnetoencephalography (MEG) data of 10 subjects were acquired during presentation. The SVM and CNN in EEGNet architecture classifiers were used to detect the onsets of target.

Results: We obtained fairly high target detection scores using SVM and EEGNet classifiers based on MEG data. The proposed ternary classification method showed that the near-non-target samples can be discriminated from others, and the separation significantly increased the ERP detection scores in the EEGNet classifier. Moreover, the visualization of the new method suggested the different underling of SVM and EEGNet classifiers in ERP detection of the RSVP experiment.

Conclusion: In the RSVP experiment, the near-non-target samples contain separable ERP activity. The ERP detection scores can be increased using classifiers of the EEGNet model, by separating the non-target into near- and far-targets based on their delay against targets.

Machine Learning-Based Electroencephalographic Phenotypes of Schizophrenia and Major Depressive Disorder

Background: Psychiatric diagnosis is formulated by symptomatic classification; disease-specific neurophysiological phenotyping could help with its fundamental treatment. Here, we investigated brain phenotyping in patients with schizophrenia (SZ) and major depressive disorder (MDD) by using electroencephalography (EEG) and conducted machine-learning-based classification of the two diseases by using EEG components. Materials and Methods: We enrolled healthy controls (HCs) (n = 30) and patients with SZ (n = 34) and MDD (n = 33). An auditory P300 (AP300) task was performed, and the N1 and P3 components were extracted. Two-group classification was conducted using linear discriminant analysis (LDA) and support vector machine (SVM) classifiers. Positive and negative symptoms and depression and/or anxiety symptoms were evaluated. Results: Considering both the results of statistical comparisons and machine learning-based classifications, patients and HCs showed significant differences in AP300, with SZ and MDD showing lower N1 and P3 than HCs. In the sum of amplitudes and cortical sources, the findings for LDA with classification accuracy (SZ vs. HCs: 71.31%, MDD vs. HCs: 74.55%), sensitivity (SZ vs. HCs: 77.67%, MDD vs. HCs: 79.00%), and specificity (SZ vs. HCs: 64.00%, MDD vs. HCs: 69.67%) supported these results. The SVM classifier showed reasonable scores between SZ and HCs and/or MDD and HCs. The comparison between SZ and MDD showed low classification accuracy (59.71%), sensitivity (65.08%), and specificity (54.83%). Conclusions: Patients with SZ and MDD showed deficiencies in N1 and P3 components in the sum of amplitudes and cortical sources, indicating attentional dysfunction in both early and late sensory/cognitive gating input. The LDA and SVM classifiers in the AP300 are useful to distinguish patients with SZ and HCs and/or MDD and HCs.

Single-Option P300-BCI Performance Is Affected by Visual Stimulation Conditions

The P300 paradigm is one of the most promising techniques for its robustness and reliability in Brain-Computer Interface (BCI) applications, but it is not exempt from shortcomings. The present work studied single-trial classification effectiveness in distinguishing between target and non-target responses considering two conditions of visual stimulation and the variation of the number of symbols presented to the user in a single-option visual frame. In addition, we also investigated the relationship between the classification results of target and non-target events when training and testing the machine-learning model with datasets containing different stimulation conditions and different number of symbols. To this end, we designed a P300 experimental protocol considering, as conditions of stimulation: the color highlighting or the superimposing of a cartoon face and from four to nine options. These experiments were carried out with 19 healthy subjects in 3 sessions. The results showed that the Event-Related Potentials (ERP) responses and the classification accuracy are stronger with cartoon faces as stimulus type and similar irrespective of the amount of options. In addition, the classification performance is reduced when using datasets with different type of stimulus, but it is similar when using datasets with different the number of symbols. These results have a special connotation for the design of systems, in which it is intended to elicit higher levels of evoked potentials and, at the same time, optimize training time.

Improving the Cross-Subject Performance of the ERP-Based Brain-Computer Interface Using Rapid Serial Visual Presentation and Correlation Analysis Rank

The brain–computer interface (BCI) is a system that is designed to provide communication channels to anyone through a computer. Initially, it was suggested to help the disabled, but actually had been proposed a wider range of applications. However, the cross-subject recognition in BCI systems is difficult to break apart from the individual specific characteristics, unsteady characteristics, and environmental specific characteristics, which also makes it difficult to develop highly reliable and highly stable BCI systems. Rapid serial visual presentation (RSVP) is one of the most recent spellers with a clean, unified background and a single stimulus, which may evoke event-related potential (ERP) patterns with less individual difference. In order to build a BCI system that allows new users to use it directly without calibration or with less calibration time, RSVP was employed as evoked paradigm, then correlation analysis rank (CAR) algorithm was proposed to improve the cross-individual classification and simultaneously use as less training data as possible. Fifty-eight subjects took part in the experiments. The flash stimulation time is 200 ms, and the off time is 100 ms. The P300 component was locked to the target representation by time. The results showed that RSVP could evoke more similar ERP patterns among subjects compared with matrix paradigm. Then, the included angle cosine was calculated and counted for averaged ERP waveform between each two subjects. The average matching number of all subjects was 6 for the matrix paradigm, while for the RSVP paradigm, the average matching number range was 20 when the threshold value was set to 0.5, more than three times as much larger, quantificationally indicating that ERP waveforms evoked by the RSVP paradigm produced smaller individual differences, and it is more favorable for cross-subject classification. Information transfer rates (ITR) were also calculated for RSVP and matrix paradigms, and the RSVP paradigm got the average ITR of 43.18 bits/min, which was 13% higher than the matrix paradigm. Then, the receiver operating characteristic (ROC) curve value was computed and compared using the proposed CAR algorithm and traditional random selection. The results showed that the proposed CAR got significantly better performance than the traditional random selection and got the best AUC value of 0.8, while the traditional random selection only achieved 0.65. These encouraging results suggest that with proper evoked paradigm and classification methods, it is feasible to get stable performance across subjects for ERP-based BCI. Thus, our findings provide a new approach to improve BCI performances.

Prediction of Event Related Potential Speller Performance Using Resting-State EEG

Event-related potential (ERP) speller can be utilized in device control and communication for locked-in or severely injured patients. However, problems such as inter-subject performance instability and ERP-illiteracy are still unresolved. Therefore, it is necessary to predict classification performance before performing an ERP speller in order to use it efficiently. In this study, we investigated the correlations with ERP speller performance using a resting-state before an ERP speller. In specific, we used spectral power and functional connectivity according to four brain regions and five frequency bands. As a result, the delta power in the frontal region and functional connectivity in the delta, alpha, gamma bands are significantly correlated with the ERP speller performance. Also, we predicted the ERP speller performance using EEG features in the resting-state. These findings may contribute to investigating the ERP-illiteracy and considering the appropriate alternatives for each user.