MEDUSA©: A novel Python-based software ecosystem to accelerate brain-computer interface and cognitive neuroscience research

Controlling Virtual Reality With Brain Signals: State of the Art of Using VR-Based Feedback in Neurofeedback Applications

The rapid progress of commercial virtual reality (VR) technology, open access to VR development software as well as open-source instructions for creating brain-VR interfaces have increased the number of VR-based neurofeedback (NF) training studies. Controlling a VR environment with brain signals has potential advantages for NF applications. More entertaining, multimodal and adaptive virtual feedback modalities might positively affect subjective user experience and could consequently enhance NF training performance and outcome. Nevertheless, there are certain pitfalls and contraindications that make VR-based NF not suitable for everyone. In the present review, we summarize applications of VR-based NF and discuss positive effects of VR-based NF training as well as contraindications such as cybersickness in VR or age- and sex-related differences. The existing literature implies that VR-based feedback is a promising tool for the improvement of NF training performance. Users generally rate VR-based feedback more positively than traditional 2D feedback, albeit to draw meaningful conclusions and to rule out adverse effects of VR, more research on this topic is necessary. The pace in the development of brain-VR synchronization furthermore necessitates ethical considerations on these technologies.

Advancements in brain-computer interfaces for the rehabilitation of unilateral spatial neglect: a concise review

This short review examines recent advancements in neurotechnologies within the context of managing unilateral spatial neglect (USN), a common condition following stroke. Despite the success of brain-computer interfaces (BCIs) in restoring motor function, there is a notable absence of effective BCI devices for treating cerebral visual impairments, a prevalent consequence of brain lesions that significantly hinders rehabilitation. This review analyzes current non-invasive BCIs and technological solutions dedicated to cognitive rehabilitation, with a focus on visuo-attentional disorders. We emphasize the need for further research into the use of BCIs for managing cognitive impairments and propose a new potential solution for USN rehabilitation, by combining the clinical subtleties of this syndrome with the technological advancements made in the field of neurotechnologies.

Unraveling motor imagery brain patterns using explainable artificial intelligence based on Shapley values

BACKGROUND AND OBJECTIVE

Motor imagery (MI) based brain-computer interfaces (BCIs) are widely used in rehabilitation due to the close relationship that exists between MI and motor execution (ME). However, the underlying brain mechanisms of MI remain not well understood. Most MI-BCIs use the sensorimotor rhythms elicited in the primary motor cortex (M1) and somatosensory cortex (S1), which consist of an event-related desynchronization followed by an event-related synchronization. Consequently, this has resulted in systems that only record signals around M1 and S1. However, MI could involve a more complex network including sensory, association, and motor areas. In this study, we hypothesize that the superior accuracies achieved by new deep learning (DL) models applied to MI decoding rely on focusing on a broader MI activation of the brain. Parallel to the success of DL, the field of explainable artificial intelligence (XAI) has seen continuous development to provide explanations for DL networks success. The goal of this study is to use XAI in combination with DL to extract information about MI brain activation patterns from non-invasive electroencephalography (EEG) signals.

METHODS

We applied an adaptation of Shapley additive explanations (SHAP) to EEGSym, a state-of-the-art DL network with exceptional transfer learning capabilities for inter-subject MI classification. We obtained the SHAP values from two public databases comprising 171 users generating left and right hand MI instances with and without real-time feedback.

RESULTS

We found that EEGSym based most of its prediction on the signal of the frontal electrodes, i.e. F7 and F8, and on the first 1500 ms of the analyzed imagination period. We also found that MI involves a broad network not only based on M1 and S1, but also on the prefrontal cortex (PFC) and the posterior parietal cortex (PPC). We further applied this knowledge to select a 8-electrode configuration that reached inter-subject accuracies of 86.5% ± 10.6% on the Physionet dataset and 88.7% ± 7.0% on the Carnegie Mellon University's dataset.

CONCLUSION

Our results demonstrate the potential of combining DL and SHAP-based XAI to unravel the brain network involved in producing MI. Furthermore, SHAP values can optimize the requirements for out-of-laboratory BCI applications involving real users.

Influence of spatial frequency in visual stimuli for cVEP-based BCIs: evaluation of performance and user experience

Code-modulated visual evoked potentials (c-VEPs) are an innovative control signal utilized in brain-computer interfaces (BCIs) with promising performance. Prior studies on steady-state visual evoked potentials (SSVEPs) have indicated that the spatial frequency of checkerboard-like stimuli influences both performance and user experience. Spatial frequency refers to the dimensions of the individual squares comprising the visual stimulus, quantified in cycles (i.e., number of black-white squares pairs) per degree of visual angle. However, the specific effects of this parameter on c-VEP-based BCIs remain unexplored. Therefore, the objective of this study is to investigate the role of spatial frequency of checkerboard-like visual stimuli in a c-VEP-based BCI. Sixteen participants evaluated selection matrices with eight spatial frequencies: C001 (0 c/°, 1×1 squares), C002 (0.15 c/°, 2×2 squares), C004 (0.3 c/°, 4×4 squares), C008 (0.6 c/°, 8×8 squares), C016 (1.2 c/°, 16×16 squares), C032 (2.4 c/°, 32×32 squares), C064 (4.79 c/°, 64×64 squares), and C128 (9.58 c/°, 128×128 squares). These conditions were tested in an online spelling task, which consisted of 18 trials each conducted on a 3×3 command interface. In addition to accuracy and information transfer rate (ITR), subjective measures regarding comfort, ocular irritation, and satisfaction were collected. Significant differences in performance and comfort were observed based on different stimulus spatial frequencies. Although all conditions achieved mean accuracy over 95% after 2.1 s of trial duration, C016 stood out in terms user experience. The proposed condition not only achieved a mean accuracy of 96.53% and 164.54 bits/min with a trial duration of 1.05s, but also was reported to be significantly more comfortable than the traditional C001 stimulus. Since both features are key for BCI development, higher spatial frequencies than the classical black-to-white stimulus might be more adequate for c-VEP systems. Hence, we assert that the spatial frequency should be carefully considered in the development of future applications for c-VEP-based BCIs.

An explainable deep-learning model to stage sleep states in children and propose novel EEG-related patterns in sleep apnea

Automatic deep-learning models used for sleep scoring in children with obstructive sleep apnea (OSA) are perceived as black boxes, limiting their implementation in clinical settings. Accordingly, we aimed to develop an accurate and interpretable deep-learning model for sleep staging in children using single-channel electroencephalogram (EEG) recordings. We used EEG signals from the Childhood Adenotonsillectomy Trial (CHAT) dataset (n = 1637) and a clinical sleep database (n = 980). Three distinct deep-learning architectures were explored to automatically classify sleep stages from a single-channel EEG data. Gradient-weighted Class Activation Mapping (Grad-CAM), an explainable artificial intelligence (XAI) algorithm, was then applied to provide an interpretation of the singular EEG patterns contributing to each predicted sleep stage. Among the tested architectures, a standard convolutional neural network (CNN) demonstrated the highest performance for automated sleep stage detection in the CHAT test set (accuracy = 86.9% and five-class kappa = 0.827). Furthermore, the CNN-based estimation of total sleep time exhibited strong agreement in the clinical dataset (intra-class correlation coefficient = 0.772). Our XAI approach using Grad-CAM effectively highlighted the EEG features associated with each sleep stage, emphasizing their influence on the CNN's decision-making process in both datasets. Grad-CAM heatmaps also allowed to identify and analyze epochs within a recording with a highly likelihood to be misclassified, revealing mixed features from different sleep stages within these epochs. Finally, Grad-CAM heatmaps unveiled novel features contributing to sleep scoring using a single EEG channel. Consequently, integrating an explainable CNN-based deep-learning model in the clinical environment could enable automatic sleep staging in pediatric sleep apnea tests.

Combining brain-computer interfaces and multiplayer video games: an application based on c-VEPs

Introduction and objective

Video games are crucial to the entertainment industry, nonetheless they can be challenging to access for those with severe motor disabilities. Brain-computer interfaces (BCI) systems have the potential to help these individuals by allowing them to control video games using their brain signals. Furthermore, multiplayer BCI-based video games may provide valuable insights into how competitiveness or motivation affects the control of these interfaces. Despite the recent advancement in the development of code-modulated visual evoked potentials (c-VEPs) as control signals for high-performance BCIs, to the best of our knowledge, no studies have been conducted to develop a BCI-driven video game utilizing c-VEPs. However, c-VEPs could enhance user experience as an alternative method. Thus, the main goal of this work was to design, develop, and evaluate a version of the well-known 'Connect 4' video game using a c-VEP-based BCI, allowing 2 users to compete by aligning 4 same-colored coins vertically, horizontally or diagonally.

Methods

The proposed application consists of a multiplayer video game controlled by a real-time BCI system processing 2 electroencephalograms (EEGs) sequentially. To detect user intention, columns in which the coin can be placed was encoded with shifted versions of a pseudorandom binary code, following a traditional circular shifting c-VEP paradigm. To analyze the usability of our application, the experimental protocol comprised an evaluation session by 22 healthy users. Firstly, each user had to perform individual tasks. Afterward, users were matched and the application was used in competitive mode. This was done to assess the accuracy and speed of selection. On the other hand, qualitative data on satisfaction and usability were collected through questionnaires.

Results

The average accuracy achieved was 93.74% ± 1.71%, using 5.25 seconds per selection. The questionnaires showed that users felt a minimal workload. Likewise, high satisfaction values were obtained, highlighting that the application was intuitive and responds quickly and smoothly.

Conclusions

This c-VEP based multiplayer video game has reached suitable performance on 22 users, supported by high motivation and minimal workload. Consequently, compared to other versions of "Connect 4" that utilized different control signals, this version has exhibited superior performance.

Nonparametric Early Stopping Detection for c-VEP-based Brain-Computer Interfaces: A Pilot Study

Brain-computer interface (BCI) systems based on code-modulated visual evoked potentials (c-VEP) stand out for achieving excellent command selection accuracies with very short calibration times. One of the natural steps to democratize their use in plug-and-play environments is to develop early stopping algorithms. These methods allow real-time detection of the minimum number of code repetitions needed to provide reliable selections. However, such techniques are scarce in the current state-of-the-art for c-VEP-based BCI systems based on the classical circular shifting paradigm. Here, a novel nonparametric early stopping method is proposed, which approximates the distribution of unattended commands to a normal distribution and issues a selection when the correlation of the command is considered an outlier. The proposal has been evaluated offline with 15 healthy users, achieving an average accuracy of 97.08% and a speed of 1.37 s/command. Likewise, the algorithm has also been evaluated with an additional user in an online way, as a proof of concept to validate its technical feasibility, achieving an average accuracy of 96.88% with a speed of 1.67 s/command. These results suggest that the real time application of the proposed algorithm is feasible, significantly reducing the required selection time without compromising accuracy.

Assessing the Potential of Brain-Computer Interface Multiplayer Video Games using c-VEPs: A Pilot Study

Video games have become a common and widespread form of entertainment, while non-invasive brain-computer interfaces (BCI) are emerging as potential alternative communication technologies. Combining BCIs and video games can enhance the gaming experience and make it accessible to motor-disabled individuals. Recently, code-modulated visual evoked potentials (c-VEP) have been proposed as a novel control signal able to achieve high performance with short calibration times. However, there are still no video games that use c-VEPs as a control signal. The aim of this pilot study is to develop an implementation of the 'Connect 4' multiplayer video game using a c-VEP-based BCI and test it with 10 healthy users. Participants were paired to compete in matches and carried out individual tasks. The results showed that the participants were able to control the game with an average accuracy of 94.10% and a selection time of 5.25 seconds per command, outperforming previous approaches. This suggests that the proposed video game is feasible and c-VEPs can provide smooth BCI control.

ITACA: An open-source framework for Neurofeedback based on Brain-Computer Interfaces

BACKGROUND AND OBJECTIVE

Neurofeedback (NF) is a paradigm that allows users to self-modulate patterns of brain activity. It is implemented with a closed-loop brain-computer interface (BCI) system that analyzes the user's brain activity in real-time and provides continuous feedback. This paradigm is of great interest due to its potential as a non-pharmacological and non-invasive alternative to treat non-degenerative brain disorders. Nevertheless, currently available NF frameworks have several limitations, such as the lack of a wide variety of real-time analysis metrics or overly simple training scenarios that may negatively affect user performance. To overcome these limitations, this work proposes ITACA: a novel open-source framework for the design, implementation and evaluation of NF training paradigms.

METHODS

ITACA is designed to be easy-to-use, flexible and attractive. Specifically, ITACA includes three different gamified training scenarios with a choice of five brain activity metrics as real-time feedback. Among them, novel metrics based on functional connectivity and network theory stand out. It is complemented with five different computerized versions of widespread cognitive assessment tests. To validate the proposed framework, a computational efficiency analysis and an NF training protocol focused on frontal-medial theta modulation were conducted.

RESULTS

Efficiency analysis proved that all implemented metrics allow an optimal feedback update rate for conducting NF sessions. Furthermore, conducted NF protocol yielded results that support the use of ITACA in NF research studies.

CONCLUSIONS

ITACA implements a wide variety of features for designing, conducting and evaluating NF studies with the goal of helping researchers expand the current state-of-the-art in NF training.