A Novel Audiovisual P300-Speller Paradigm Based on Cross-Modal Spatial and Semantic Congruence

Brain-computer interface in critical care and rehabilitation

This comprehensive review explores the broad landscape of brain-computer interface (BCI) technology and its potential use in intensive care units (ICUs), particularly for patients with motor impairments such as quadriplegia or severe brain injury. By employing brain signals from various sensing techniques, BCIs offer enhanced communication and motor rehabilitation strategies for patients. This review underscores the concept and efficacy of noninvasive, electroencephalogram-based BCIs in facilitating both communicative interactions and motor function recovery. Additionally, it highlights the current research gap in intuitive "stop" mechanisms within motor rehabilitation protocols, emphasizing the need for advancements that prioritize patient safety and individualized responsiveness. Furthermore, it advocates for more focused research that considers the unique requirements of ICU environments to address the challenges arising from patient variability, fatigue, and limited applicability of current BCI systems outside of experimental settings.

Non-spatial inhibition of return attenuates audiovisual integration owing to modality disparities

Although previous studies have investigated the relationship between inhibition of return (IOR) and multisensory integration, the influence of non-spatial has not been explored. The present study aimed to investigate the influence of non-spatial IOR on audiovisual integration by using a "prime-neutral cue-target" paradigm. In Experiment 1, which manipulated prime validity and target modality, the targets were positioned centrally, revealing significant non-spatial IOR effects in the visual, auditory, and audiovisual modalities. Analysis of relative multisensory response enhancement (rMRE) indicated substantial audiovisual integration enhancement in both valid and invalid target conditions. Furthermore, the enhancement was weaker for valid targets than for invalid targets. In Experiment 2, the targets were positioned above and below to rule out repetition blindness (RB); this experiment successfully replicated the results observed in Experiment 1. Notably, Experiments 1 and 2 consistently found that the correlation between modality differences and rMRE for valid targets indicated that differences in signal strength between visual and auditory modalities contributed to a reduction in audiovisual integration. However, the absence of correlation with the invalid target suggests that attention, as a key factor, may play a significant role in this process. The present study highlights how non-spatial IOR reduces audiovisual integration and sheds light on the complex interaction between attention and multisensory integration.

Advances in P300 brain-computer interface spellers: toward paradigm design and performance evaluation

A brain-computer interface (BCI) is a non-muscular communication technology that provides an information exchange channel for our brains and external devices. During the decades, BCI has made noticeable progress and has been applied in many fields. One of the most traditional BCI applications is the BCI speller. This article primarily discusses the progress of research into P300 BCI spellers and reviews four types of P300 spellers: single-modal P300 spellers, P300 spellers based on multiple brain patterns, P300 spellers with multisensory stimuli, and P300 spellers with multiple intelligent techniques. For each type of P300 speller, we further review several representative P300 spellers, including their design principles, paradigms, algorithms, experimental performance, and corresponding advantages. We particularly emphasized the paradigm design ideas, including the overall layout, individual symbol shapes and stimulus forms. Furthermore, several important issues and research guidance for the P300 speller were identified. We hope that this review can assist researchers in learning the new ideas of these novel P300 spellers and enhance their practical application capability.

A novel brain-computer interface based on audio-assisted visual evoked EEG and spatial-temporal attention CNN

Objective

Brain-computer interface (BCI) can translate intentions directly into instructions and greatly improve the interaction experience for disabled people or some specific interactive applications. To improve the efficiency of BCI, the objective of this study is to explore the feasibility of an audio-assisted visual BCI speller and a deep learning-based single-trial event related potentials (ERP) decoding strategy.

Approach

In this study, a two-stage BCI speller combining the motion-onset visual evoked potential (mVEP) and semantically congruent audio evoked ERP was designed to output the target characters. In the first stage, the different group of characters were presented in the different locations of visual field simultaneously and the stimuli were coded to the mVEP based on a new space division multiple access scheme. And then, the target character can be output based on the audio-assisted mVEP in the second stage. Meanwhile, a spatial-temporal attention-based convolutional neural network (STA-CNN) was proposed to recognize the single-trial ERP components. The CNN can learn 2-dimentional features including the spatial information of different activated channels and time dependence among ERP components. In addition, the STA mechanism can enhance the discriminative event-related features by adaptively learning probability weights.

Main results

The performance of the proposed two-stage audio-assisted visual BCI paradigm and STA-CNN model was evaluated using the Electroencephalogram (EEG) recorded from 10 subjects. The average classification accuracy of proposed STA-CNN can reach 59.6 and 77.7% for the first and second stages, which were always significantly higher than those of the comparison methods (p < 0.05).

Significance

The proposed two-stage audio-assisted visual paradigm showed a great potential to be used to BCI speller. Moreover, through the analysis of the attention weights from time sequence and spatial topographies, it was proved that STA-CNN could effectively extract interpretable spatiotemporal EEG features.

Visuo-auditory stimuli with semantic, temporal and spatial congruence for a P300-based BCI: An exploratory test with an ALS patient in a completely locked-in state

BACKGROUND

Brain-computer interfaces (BCIs) are a promising tool for communication with completely locked-in state (CLIS) patients. Despite the great efforts already made by the BCI research community, the cases of success are still very few, very exploratory, limited in time, and based on simple 'yes/no' paradigms.

NEW METHOD

A P300-based BCI is proposed comparing two conditions, one corresponding to purely spatial auditory stimuli (AU-S) and the other corresponding to hybrid visual and spatial auditory stimuli (HVA-S). In the HVA-S condition, there is a semantic, temporal, and spatial congruence between visual and auditory stimuli. The stimuli comprise a lexicon of 7 written and spoken words. Spatial sounds are generated through the head-related transfer function. Given the good results obtained with 10 able-bodied participants, we investigated whether a patient entering CLIS could use the proposed BCI.

RESULTS

The able-bodied group achieved 71.3 % and 90.5 % online classification accuracy for the auditory and hybrid BCIs respectively, while the patient achieved 30 % and chance level accuracies, for the same conditions. Notwithstanding, the patient's event-related potentials (ERPs) showed statistical discrimination between target and non-target events in different time windows.

COMPARISON WITH EXISTING METHODS

The results of the control group compare favorably with the state-of-the-art, considering a 7-class BCI controlled visual-covertly and with auditory stimuli. The integration of visual and auditory stimuli has not been tested before with CLIS patients.

CONCLUSIONS

The semantic, temporal, and spatial congruence of the stimuli increased the performance of the control group, but not of the CLIS patient, which can be due to impaired attention and cognitive function. The patient's unique ERP patterns make interpretation difficult, requiring further tests/paradigms to decouple patients' responses at different levels (reflexive, perceptual, cognitive). The ERPs discrimination found indicates that a simplification of the proposed approaches may be feasible.

A P300-Detection Method Based on Logistic Regression and a Convolutional Neural Network

Background

Electroencephalogram (EEG)-based brain-computer interface (BCI) systems are widely utilized in various fields, including health care, intelligent assistance, identity recognition, emotion recognition, and fatigue detection. P300, the main event-related potential, is the primary component detected by EEG-based BCI systems. Existing algorithms for P300 classification in EEG data usually perform well when tested in a single participant, although they exhibit significant decreases in accuracy when tested in new participants. We attempted to address this lack of generalizability associated with existing classification methods using a novel convolutional neural network (CNN) model developed using logistic regression (LR).

Materials and Methods

We proposed an LR-CNN model comprising two parts: a combined LR-based memory model and a CNN-based generalization model. The LR-based memory model can learn the individual features of participants and addresses the decrease in accuracy caused by individual differences when applied to new participants. The CNN-based generalization model can learn the common features among participants, thereby reducing overall classification bias and improving overall classification accuracy.

Results

We compared our method with existing, commonly used classification methods through three different sets of experiments. The experimental results indicated that our method could learn individual differences among participants. Compared with other commonly used classification methods, our method yielded a marked improvement (>90%) in classification among new participants.

Conclusion

The accuracy of the proposed model in the face of new participants is better than that of existing, commonly used classification methods. Such improvements in cross-subject test accuracy will aid in the development of BCI systems.

Decoding Covert Speech From EEG-A Comprehensive Review

Over the past decade, many researchers have come up with different implementations of systems for decoding covert or imagined speech from EEG (electroencephalogram). They differ from each other in several aspects, from data acquisition to machine learning algorithms, due to which, a comparison between different implementations is often difficult. This review article puts together all the relevant works published in the last decade on decoding imagined speech from EEG into a single framework. Every important aspect of designing such a system, such as selection of words to be imagined, number of electrodes to be recorded, temporal and spatial filtering, feature extraction and classifier are reviewed. This helps a researcher to compare the relative merits and demerits of the different approaches and choose the one that is most optimal. Speech being the most natural form of communication which human beings acquire even without formal education, imagined speech is an ideal choice of prompt for evoking brain activity patterns for a BCI (brain-computer interface) system, although the research on developing real-time (online) speech imagery based BCI systems is still in its infancy. Covert speech based BCI can help people with disabilities to improve their quality of life. It can also be used for covert communication in environments that do not support vocal communication. This paper also discusses some future directions, which will aid the deployment of speech imagery based BCI for practical applications, rather than only for laboratory experiments.

A Spelling Paradigm With an Added Red Dot Improved the P300 Speller System Performance

The traditional P300 speller system uses the flashing row or column spelling paradigm. However, the classification accuracy and information transfer rate of the P300 speller are not adequate for real-world application. To improve the performance of the P300 speller, we devised a new spelling paradigm in which the flashing row or column of a virtual character matrix is covered by a translucent green circle with a red dot in either the upper or lower half (GC-RD spelling paradigm). We compared the event-related potential (ERP) waveforms with a control paradigm (GC spelling paradigm), in which the flashing row or column of a virtual character matrix was covered by a translucent green circle only. Our experimental results showed that the amplitude of P3a at the parietal area and P3b at the frontal–central–parietal areas evoked by the GC-RD paradigm were significantly greater than those induced by the GC paradigm. Higher classification accuracy and information transmission rates were also obtained in the GC-RD system. Our results indicated that the added red dots increased attention and visuospatial information, resulting in an amplitude increase in both P3a and P3b, thereby improving the performance of the P300 speller system.

A Practical EEG-Based Human-Machine Interface to Online Control an Upper-Limb Assist Robot

Background and Objective: Electroencephalography (EEG) can be used to control machines with human intention, especially for paralyzed people in rehabilitation exercises or daily activities. Some effort was put into this but still not enough for online use. To improve the practicality, this study aims to propose an efficient control method based on P300, a special EEG component. Moreover, we have developed an upper-limb assist robot system with the method for verification and hope to really help paralyzed people.

Methods: We chose P300, which is highly available and easily accepted to obtain the user's intention. Preprocessing and spatial enhancement were firstly implemented on raw EEG data. Then, three approaches– linear discriminant analysis, support vector machine, and multilayer perceptron –were compared in detail to accomplish an efficient P300 detector, whose output was employed as a command to control the assist robot.

Results: The method we proposed achieved an accuracy of 94.43% in the offline test with the data from eight participants. It showed sufficient reliability and robustness with an accuracy of 80.83% and an information transfer rate of 15.42 in the online test. Furthermore, the extended test showed remarkable generalizability of this method that can be used in more complex application scenarios.

Conclusion: From the results, we can see that the proposed method has great potential for helping paralyzed people easily control an assist robot to do numbers of things.