Towards the automated localisation of targets in rapid image-sifting by collaborative brain-computer interfaces

Optimization of transfer learning based on source sample selection in Euclidean space for P300-based brain-computer interfaces

Introduction

Event-related potentials (ERPs), such as P300, are widely utilized for non-invasive monitoring of brain activity in brain-computer interfaces (BCIs) via electroencephalogram (EEG). However, the non-stationary nature of EEG signals and different data distributions among subjects create significant challenges for implementing real-time P300-based BCIs. This requires time-consuming calibration and a large number of training samples.

Methods

To address these challenges, this study proposes a transfer learning-based approach that uses a convolutional neural network for high-level feature extraction, followed by Euclidean space data alignment to ensure similar distributions of extracted features. Furthermore, a source selection technique based on the Euclidean distance metric was applied to measure the distance between each source feature sample and a reference point from the target domain. The samples with the lowest distance were then chosen to increase the similarity between source and target datasets. Finally, the transferred features are applied to a discriminative restricted Boltzmann machine classifier for P300 detection.

Results

The proposed method was evaluated on the state-of-the-art BCI Competition III dataset II and rapid serial visual presentation dataset. The results demonstrate that the proposed technique achieves an average accuracy of 97% for both online and offline after 15 repetitions, which is comparable to the state-of-the-art methods. Notably, the proposed approach requires <½ of the training samples needed by previous studies.

Discussion

Therefore, this technique offers an efficient solution for developing ERP-based BCIs with robust performance against reduced a number of training data.

A Cross-Scale Transformer and Triple-View Attention Based Domain-Rectified Transfer Learning for EEG Classification in RSVP Tasks

Rapid serial visual presentation (RSVP)-based brain-computer interface (BCI) is a promising target detection technique by using electroencephalogram (EEG) signals. However, existing deep learning approaches seldom considered dependencies of multi-scale temporal features and discriminative multi-view spectral features simultaneously, which limits the representation learning ability of the model and undermine the EEG classification performance. In addition, recent transfer learning-based methods generally failed to obtain transferable cross-subject invariant representations and commonly ignore the individual-specific information, leading to the poor cross-subject transfer performance. In response to these limitations, we propose a cross-scale Transformer and triple-view attention based domain-rectified transfer learning (CST-TVA-DRTL) for the RSVP classification. Specially, we first develop a cross-scale Transformer (CST) to extract multi-scale temporal features and exploit the dependencies of different scales features. Then, a triple-view attention (TVA) is designed to capture spectral features from triple views of multi-channel time-frequency images. Finally, a domain-rectified transfer learning (DRTL) framework is proposed to simultaneously obtain transferable domain-invariant representations and untransferable domain-specific representations, then utilize domain-specific information to rectify domain-invariant representations to adapt to target data. Experimental results on two public RSVP datasets suggests that our CST-TVA-DRTL outperforms the state-of-the-art methods in the RSVP classification task. The source code of our model is publicly available in https://github.com/ljbuaa/CST_TVA_DRTL.

Single-Trial EEG Classification Using Spatio-Temporal Weighting and Correlation Analysis for RSVP-Based Collaborative Brain Computer Interface

OBJECTIVE

Since single brain computer interface (BCI) is limited in performance, it is necessary to develop collaborative BCI (cBCI) systems which integrate multi-user electroencephalogram (EEG) information to improve system performance. However, there are still some challenges in cBCI systems, including effective discriminant feature extraction of multi-user EEG data, fusion algorithms, time reduction of system calibration, etc. Methods: This study proposed an event-related potential (ERP) feature extraction and classification algorithm of spatio-temporal weighting and correlation analysis (STC) to improve the performance of cBCI systems. The proposed STC algorithm consisted of three modules. First, source extraction and interval modeling were used to overcome the problem of inter-trial variability. Second, spatio-temporal weighting and temporal projection were utilized to extract effective discriminant features for multi-user information fusion and cross-session transfer. Third, correlation analysis was conducted to match target/non-target templates for classification of multi-user and cross-session datasets.

RESULTS

The collaborative cross-session datasets of rapid serial visual presentation (RSVP) from 14 subjects were used to evaluate the performance of the EEG classification algorithm. For single-user/collaborative EEG classification of within-session and cross-session datasets, STC had significantly higher performance than the existing state-of-the-art machine learning algorithms.

CONCLUSION

It was demonstrated that STC was effective to improve the classification performance of multi-user collaboration and cross-session transfer for RSVP-based BCI systems, and was helpful to reduce the system calibration time.

Decoupling Representation Learning for Imbalanced Electroencephalography Classification in Rapid Serial Visual Presentation Task

OBJECTIVE

The class imbalance problem considerably restricts the performance of electroencephalography (EEG) classification in the rapid serial visual presentation (RSVP) task. Existing solutions typically employ re-balancing strategies (e.g. re-weighting and re-sampling) to alleviate the impact of class imbalance, which enhances the classifier learning of deep networks but unexpectedly damages the representative ability of the learned deep features as original distributions become distorted.

APPROACH

In this study, a novel decoupling representation learning (DRL) model, has been proposed that separates the representation learning and classification processes to capture the discriminative feature of imbalanced RSVP EEG data while classifying it accurately. The representation learning process is responsible for learning universal patterns for the classification of all samples, while the classifier determines a better bounding for the target and non-target classes. Specifically, the representation learning process adopts a dual-branch architecture, which minimizes the contrastive loss to regularize the representation space. In addition, to learn more discriminative information from RSVP EEG data, a novel multi-granular information (MGI) based extractor is designed to extract spatial-temporal information. Considering the class re-balancing strategies can significantly promote classifier learning, the classifier was trained with rebalanced EEG data while freezing the parameters of the representation learning process.

MAIN RESULTS

To evaluate the proposed method, experiments were conducted on two public datasets and one self-conducted dataset. The results demonstrate that the proposed DRL can achieve state-of-the-art performance for EEG classification in the RSVP task.

SIGNIFICANCE

This is the first study to focus on the class imbalance problem and propose a generic solution in the RSVP task. Furthermore, multi-granular data was explored to extract more complementary spatial-temporal information. The code is open-source and available at https://github.com/Tammie-Li/DRL.

Workshops of the Eighth International Brain-Computer Interface Meeting: BCIs: The Next Frontier

The Eighth International Brain-Computer Interface (BCI) Meeting was held June 7-9th, 2021 in a virtual format. The conference continued the BCI Meeting series' interactive nature with 21 workshops covering topics in BCI (also called brain-machine interface) research. As in the past, workshops covered the breadth of topics in BCI. Some workshops provided detailed examinations of specific methods, hardware, or processes. Others focused on specific BCI applications or user groups. Several workshops continued consensus building efforts designed to create BCI standards and increase the ease of comparisons between studies and the potential for meta-analysis and large multi-site clinical trials. Ethical and translational considerations were both the primary topic for some workshops or an important secondary consideration for others. The range of BCI applications continues to expand, with more workshops focusing on approaches that can extend beyond the needs of those with physical impairments. This paper summarizes each workshop, provides background information and references for further study, presents an overview of the discussion topics, and describes the conclusion, challenges, or initiatives that resulted from the interactions and discussion at the workshop.

A Collaborative Brain-Computer Interface Framework for Enhancing Group Detection Performance of Dynamic Visual Targets

The superiority of collaborative brain-computer interface (cBCI) in performance enhancement makes it an effective way to break through the performance bottleneck of the BCI-based dynamic visual target detection. However, the existing cBCIs focus on multi-mind information fusion with a static and unidirectional mode, lacking the information interaction and learning guidance among multiple agents. Here, we propose a novel cBCI framework to enhance the group detection performance of dynamic visual targets. Specifically, a mutual learning domain adaptation network (MLDANet) with information interaction, dynamic learning, and individual transferring abilities is developed as the core of the cBCI framework. MLDANet takes P3-sSDA network as individual network unit, introduces mutual learning strategy, and establishes a dynamic interactive learning mechanism between individual networks and collaborative decision-making at the neural decision level. The results indicate that the proposed MLDANet-cBCI framework can achieve the best group detection performance, and the mutual learning strategy can improve the detection ability of individual networks. In MLDANet-cBCI, the F1 scores of collaborative detection and individual network are 0.12 and 0.19 higher than those in the multi-classifier cBCI, respectively, when three minds collaborate. Thus, the proposed framework breaks through the traditional multi-mind collaborative mode and exhibits a superior group detection performance of dynamic visual targets, which is also of great significance for the practical application of multi-mind collaboration.

Phase Preservation Neural Network for Electroencephalography Classification in Rapid Serial Visual Presentation Task

Neuroscience studies have demonstrated the phase-locked characteristics of some early event-related potential (ERP) components evoked by stimuli. In this study, we propose a phase preservation neural network (PPNN) to learn phase information to improve the Electroencephalography (EEG) classification in a rapid serial visual presentation (RSVP) task. The PPNN consists of three major modules that can produce spatial and temporal representations with the high discriminative ability of the EEG features for classification. We first adopt a stack of dilated temporal convolution layers to extract temporal dynamics while avoiding the loss of phase information. Considering the intrinsic channel dependence of the EEG data, a spatial convolution layer is then applied to obtain the spatial-temporal representation of the input EEG signal. Finally, a fully connected layer is adopted to extract higher-level features for the final classification. The experiments are conducted on two public and one collected EEG datasets from the RSVP task, in which we evaluated the performance and explored the capability of phase preservation of our PPNN model and visualized the extracted features. The experimental results indicate the superiority of the proposed PPNN when compared with previous methods, suggesting the PPNN is a robust model for EEG classification in RSVP task.

A neural network predicting the amplitude of the N2pc in individual EEG datasets

Objective.The N2pc is a small amplitude transient interhemispheric voltage asymmetry used in cognitive neuroscience to investigate subject's allocation of selective visuo-spatial attention. N2pc is typically estimated by averaging the sweeps of the electroencephalographic (EEG) signal but, in absence of explicit normative indications, the number of sweeps is often based on arbitrariness or personal experience. With the final aim of reducing duration and cost of experimental protocols, here we developed a new approach to reliably predict N2pc amplitude from a minimal EEG dataset.Approach.First, features predictive of N2pc amplitude were identified in the time-frequency domain. Then, an artificial neural network (NN) was trained to predict N2pc mean amplitude at the individual level. By resorting to simulated data, accuracy of the NN was assessed by computing the mean squared error (MSE) and the amplitude discretization error (ADE) and compared to the standard time averaging (TA) technique. The NN was then tested against two real datasets consisting of 14 and 12 subjects, respectively.Main result.In simulated scenarios entailing different number of sweeps (between 10 and 100), the MSE obtained with the proposed method resulted, on average, 1/5 of that obtained with the TA technique. Implementation on real EEG datasets showed that N2pc amplitude could be reliably predicted with as few as 40 EEG sweeps per cell of the experimental design.Significance.The developed approach allows to reduce duration and cost of experiments involving the N2pc, for instance in studies investigating attention deficits in pathological subjects.

Advanced Bioelectrical Signal Processing Methods: Past, Present and Future Approach-Part II: Brain Signals

As it was mentioned in the previous part of this work (Part I)-the advanced signal processing methods are one of the quickest and the most dynamically developing scientific areas of biomedical engineering with their increasing usage in current clinical practice. In this paper, which is a Part II work-various innovative methods for the analysis of brain bioelectrical signals were presented and compared. It also describes both classical and advanced approaches for noise contamination removal such as among the others digital adaptive and non-adaptive filtering, signal decomposition methods based on blind source separation, and wavelet transform.

Anytime collaborative brain-computer interfaces for enhancing perceptual group decision-making

In this paper we present, and test in two realistic environments, collaborative Brain-Computer Interfaces (cBCIs) that can significantly increase both the speed and the accuracy of perceptual group decision-making. The key distinguishing features of this work are: (1) our cBCIs combine behavioural, physiological and neural data in such a way as to be able to provide a group decision at any time after the quickest team member casts their vote, but the quality of a cBCI-assisted decision improves monotonically the longer the group decision can wait; (2) we apply our cBCIs to two realistic scenarios of military relevance (patrolling a dark corridor and manning an outpost at night where users need to identify any unidentified characters that appear) in which decisions are based on information conveyed through video feeds; and (3) our cBCIs exploit Event-Related Potentials (ERPs) elicited in brain activity by the appearance of potential threats but, uniquely, the appearance time is estimated automatically by the system (rather than being unrealistically provided to it). As a result of these elements, in the two test environments, groups assisted by our cBCIs make both more accurate and faster decisions than when individual decisions are integrated in more traditional manners.

Recent Approaches on Classification and Feature Extraction of EEG Signal: A Review

Objective:

Electroencephalography (EEG) has an influential role in neuroscience and commercial applications. Most of the tools available for EEG signal analysis use machine learning to extract the required information. So, the study of robust techniques for feature extraction and classification is an important thing to understand the practical use of EEG. The paper aims that if there is any special tool for a particular task. Which feature domain or classifier has a significant role in EEG signal analysis?

Approach:

It presents a detailed report of the current trend for bio-electrical signals classification focusing on various classifiers’ advantages and disadvantages. This study includes literature from 2000 to 2021 with a brief description of EEG signal origin and advancement in classification techniques.

Results:

Randomly used classifiers for EEG signal can be categorized into five classes, namely Linear Classifiers, Nearest Neighbor Classifiers, Nonlinear Bayesian Classifiers, Neural Networks, and Combinations of Classifiers. Approximately 40% of studies use Support Vector Machine, Nearest Neighbor, and their combination with others. For specific tasks, particular classifiers are recommended in the survey. Features can be defined into four categories, namely TDFs, FDFs, TFDFs, and statistical features, where 39% of studies used TFDFs. Multi-domains features are preferred when the required information cannot be obtained from one domain.

Significance:

The paper summarizes the recent approaches for feature extraction and classification of EEG signals. It describes the brain waves with their classification, related behavior, and task with the physiological correlation. The comparative analysis of different classifiers, toolbox, the channel used, accuracy, and the number of subjects from various studies can help the practitioners choose a suitable classifier. Furthermore, future directions can cope up with the relevant problems and can lead to accurate classification.

A Cross-Session Dataset for Collaborative Brain-Computer Interfaces Based on Rapid Serial Visual Presentation

Brain-computer interfaces (BCIs) based on rapid serial visual presentation (RSVP) have been widely used to categorize target and non-target images. However, it is still a challenge to detect single-trial event related potentials (ERPs) from electroencephalography (EEG) signals. Besides, the variability of EEG signal over time may cause difficulties of calibration in long-term system use. Recently, collaborative BCIs have been proposed to improve the overall BCI performance by fusing brain activities acquired from multiple subjects. For both individual and collaborative BCIs, feature extraction and classification algorithms that can be transferred across sessions can significantly facilitate system calibration. Although open datasets are highly efficient for developing algorithms, currently there is still a lack of datasets for a collaborative RSVP-based BCI. This paper presents a cross-session EEG dataset of a collaborative RSVP-based BCI system from 14 subjects, who were divided into seven groups. In collaborative BCI experiments, two subjects did the same target image detection tasks synchronously. All subjects participated in the same experiment twice with an average interval of ∼23 days. The results in data evaluation indicate that adequate signal processing algorithms can greatly enhance the cross-session BCI performance in both individual and collaborative conditions. Besides, compared with individual BCIs, the collaborative methods that fuse information from multiple subjects obtain significantly improved BCI performance. This dataset can be used for developing more efficient algorithms to enhance performance and practicality of a collaborative RSVP-based BCI system.

Thinker invariance: enabling deep neural networks for BCI across more people

OBJECTIVE

Most deep neural networks (DNNs) used as brain computer interfaces (BCI) classifiers are rarely viable for more than one person and are relatively shallow compared to the state-of-the-art in the wider machine learning literature. The goal of this work is to frame these as a unified challenge and reconsider how transfer learning is used to overcome these difficulties.

APPROACH

We present two variations of a holistic approach to transfer learning with DNNs for BCI that rely on a deeper network called TIDNet. Our approaches use multiple subjects for training in the interest of creating a more universal classifier that is applicable for new (unseen) subjects. The first approach is purely subject-invariant and the second targets specific subjects, without loss of generality. We use five publicly accessible datasets covering a range of tasks and compare our approaches to state-of-the-art alternatives in detail.

MAIN RESULTS

We observe that TIDNet in conjunction with our training augmentations is more consistent when compared to shallower baselines, and in some cases exhibits large and significant improvements, for instance motor imagery classification improvements of over 8%. Furthermore, we show that our suggested multi-domain learning (MDL) strategy strongly outperforms simply fine-tuned general models when targeting specific subjects, while remaining more generalizable to still unseen subjects.

SIGNIFICANCE

TIDNet in combination with a data alignment-based training augmentation proves to be a consistent classification approach of single raw trials and can be trained even with the inclusion of corrupted trials. Our MDL strategy calls into question the intuition to fine-tune trained classifiers to new subjects, as it proves simpler and more accurate while remaining general. Furthermore, we show evidence that augmented TIDNet training makes better use of additional subjects, showing continued and greater performance improvement over shallower alternatives, indicating promise for a new subject-invariant paradigm rather than a subject-specific one.

Can a Soldier Say No to an Enhancing Intervention?

Technological advancements have provided militaries with the possibility to enhance human performance and to provide soldiers with better warfighting capabilities. Though these technologies hold significant potential, their use is not without cost to the individual. This paper explores the complexities associated with using human cognitive enhancements in the military, focusing on how the purpose and context of these technologies could potentially undermine a soldier’s ability to say no to these interventions. We focus on cognitive enhancements and their ability to also enhance a soldier’s autonomy (i.e., autonomy-enhancing technologies). Through this lens, we explore situations that could potentially compel a soldier to accept such technologies and how this acceptance could impact rights to individual autonomy and informed consent within the military. In this examination, we highlight the contextual elements of vulnerability—institutional and differential vulnerability. In addition, we focus on scenarios in which a soldier’s right to say no to such enhancements can be diminished given the special nature of their work and the significance of making better moral decisions. We propose that though in some situations, a soldier may be compelled to accept said enhancements; with their right to say no diminished, it is not a blanket rule, and safeguards ought to be in place to ensure that autonomy and informed consent are not overridden.

Transfer Learning for Brain-Computer Interfaces: A Euclidean Space Data Alignment Approach

OBJECTIVE

This paper targets a major challenge in developing practical electroencephalogram (EEG)-based brain-computer interfaces (BCIs): how to cope with individual differences so that better learning performance can be obtained for a new subject, with minimum or even no subject-specific data?

METHODS

We propose a novel approach to align EEG trials from different subjects in the Euclidean space to make them more similar, and hence improve the learning performance for a new subject. Our approach has three desirable properties: first, it aligns the EEG trials directly in the Euclidean space, and any signal processing, feature extraction, and machine learning algorithms can then be applied to the aligned trials; second, its computational cost is very low; and third, it is unsupervised and does not need any label information from the new subject.

RESULTS

Both offline and simulated online experiments on motor imagery classification and event-related potential classification verified that our proposed approach outperformed a state-of-the-art Riemannian space data alignment approach, and several approaches without data alignment.

CONCLUSION

The proposed Euclidean space EEG data alignment approach can greatly facilitate transfer learning in BCIs.

SIGNIFICANCE

Our proposed approach is effective, efficient, and easy to implement. It could be an essential pre-processing step for EEG-based BCIs.

Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects

Recent advances in neuroscience have paved the way to innovative applications that cognitively augment and enhance humans in a variety of contexts. This paper aims at providing a snapshot of the current state of the art and a motivated forecast of the most likely developments in the next two decades. Firstly, we survey the main neuroscience technologies for both observing and influencing brain activity, which are necessary ingredients for human cognitive augmentation. We also compare and contrast such technologies, as their individual characteristics (e.g., spatio-temporal resolution, invasiveness, portability, energy requirements, and cost) influence their current and future role in human cognitive augmentation. Secondly, we chart the state of the art on neurotechnologies for human cognitive augmentation, keeping an eye both on the applications that already exist and those that are emerging or are likely to emerge in the next two decades. Particularly, we consider applications in the areas of communication, cognitive enhancement, memory, attention monitoring/enhancement, situation awareness and complex problem solving, and we look at what fraction of the population might benefit from such technologies and at the demands they impose in terms of user training. Thirdly, we briefly review the ethical issues associated with current neuroscience technologies. These are important because they may differentially influence both present and future research on (and adoption of) neurotechnologies for human cognitive augmentation: an inferior technology with no significant ethical issues may thrive while a superior technology causing widespread ethical concerns may end up being outlawed. Finally, based on the lessons learned in our analysis, using past trends and considering other related forecasts, we attempt to forecast the most likely future developments of neuroscience technology for human cognitive augmentation and provide informed recommendations for promising future research and exploitation avenues.

Bridging Waves and Crucial Events in the Dynamics of the Brain

Earlier research work on the dynamics of the brain, disclosing the existence of crucial events, is revisited for the purpose of making the action of crucial events, responsible for the 1/f −noise in the brain, compatible with the wave-like nature of the brain processes. We review the relevant neurophysiological literature to make clear that crucial events are generated by criticality. We also show that although criticality generates a strong deviation from the regular wave-like behavior, under the form of Rapid Transition Processes, the brain dynamics also host crucial events in regions of nearly coherent oscillations, thereby making many crucial events virtually invisible. Furthermore, the anomalous scaling generated by the crucial events can be established with high accuracy by means of direct analysis of raw data, suggested by a theoretical perspective not requiring the crucial events to yield a visible physical effect. The latter follows from the fact that periodicity, waves and crucial events are the consequences of a spontaneous process of self-organization. We obtain three main results: (a) the important role of crucial events is confirmed and established with greater accuracy than previously; (b) we demonstrate the theoretical tools necessary to understand the joint action of crucial events and periodicity; (c) we argue that the results of this paper can be used to shed light on the nature of this important process of self-organization, thereby contributing to the understanding of cognition.