A review of brain-computer interface games and an opinion survey from researchers, developers and users

Enhancing learning experiences: EEG-based passive BCI system adapts learning speed to cognitive load in real-time, with motivation as catalyst

Computer-based learning has gained popularity in recent years, providing learners greater flexibility and freedom. However, these learning environments do not consider the learner's mental state in real-time, resulting in less optimized learning experiences. This research aimed to explore the effect on the learning experience of a novel EEG-based Brain-Computer Interface (BCI) that adjusts the speed of information presentation in real-time during a learning task according to the learner's cognitive load. We also explored how motivation moderated these effects. In accordance with three experimental groups (non-adaptive, adaptive, and adaptive with motivation), participants performed a calibration task (n-back), followed by a memory-based learning task concerning astrological constellations. Learning gains were assessed based on performance on the learning task. Self-perceived mental workload, cognitive absorption and satisfaction were assessed using a post-test questionnaire. Between-group analyses using Mann-Whitney tests suggested that combining BCI and motivational factors led to more significant learning gains and an improved learning experience. No significant difference existed between the BCI without motivational factor and regular non-adaptive interface for overall learning gains, self-perceived mental workload, and cognitive absorption. However, participants who undertook the experiment with an imposed learning pace reported higher overall satisfaction with their learning experience and a higher level of temporal stress. Our findings suggest BCI's potential applicability and feasibility in improving memorization-based learning experiences. Further work should seek to optimize the BCI adaptive index and explore generalizability to other learning contexts.

Application of hybrid SSVEP + P300 brain computer interface to control avatar movement in mobile virtual reality gaming environment

INTRODUCTION

This research evaluated the feasibility of a hybrid SSVEP + P300 brain computer interface (BCI) for controlling the movement of an avatar in a virtual reality (VR) gaming environment (VR + BCI). Existing VR + BCI gaming environments have limitations, such as visual fatigue, a lower communication rate, minimum accuracy, and poor system comfort. Hence, there is a need for an optimized hybrid BCI system that can simultaneously evoke the strongest P300 and SSVEP potentials in the cortex.

METHODS

A BCI headset was coupled with a VR headset to generate a VR + BCI environment. The author developed a VR game in which the avatar's movement is controlled using the user's cortical responses with the help of a BCI headset. Specifically designed visual stimuli were used in the proposed system to elicit the strongest possible responses from the user's brain. The proposed system also includes an auditory feedback mechanism to facilitate precise avatar movement.

RESULTS AND CONCLUSIONS

Conventional P300 BCI and SSVEP BCI were also used to control the movements of the avatar, and their performance metrics were compared to those of the proposed system. The results demonstrated that the hybrid SSVEP + P300 BCI system was superior to the other systems for controlling avatar movement.

Evaluating the Feasibility of Visual Imagery for an EEG-Based Brain-Computer Interface

Visual imagery, or the mental simulation of visual information from memory, could serve as an effective control paradigm for a brain-computer interface (BCI) due to its ability to directly convey the user's intention with many natural ways of envisioning an intended action. However, multiple initial investigations into using visual imagery as a BCI control strategies have been unable to fully evaluate the capabilities of true spontaneous visual mental imagery. One major limitation in these prior works is that the target image is typically displayed immediately preceding the imagery period. This paradigm does not capture spontaneous mental imagery as would be necessary in an actual BCI application but something more akin to short-term retention in visual working memory. Results from the present study show that short-term visual imagery following the presentation of a specific target image provides a stronger, more easily classifiable neural signature in EEG than spontaneous visual imagery from long-term memory following an auditory cue for the image. We also show that short-term visual imagery and visual perception share commonalities in the most predictive electrodes and spectral features. However, visual imagery received greater influence from frontal electrodes whereas perception was mostly confined to occipital electrodes. This suggests that visual perception is primarily driven by sensory information whereas visual imagery has greater contributions from areas associated with memory and attention. This work provides the first direct comparison of short-term and long-term visual imagery tasks and provides greater insight into the feasibility of using visual imagery as a BCI control strategy.

Personalized Deep Learning for Substance Use in Hawaii: Protocol for a Passive Sensing and Ecological Momentary Assessment Study

BACKGROUND

Artificial intelligence (AI)-powered digital therapies that detect methamphetamine cravings via consumer devices have the potential to reduce health care disparities by providing remote and accessible care solutions to communities with limited care solutions, such as Native Hawaiian, Filipino, and Pacific Islander communities. However, Native Hawaiian, Filipino, and Pacific Islander communities are understudied with respect to digital therapeutics and AI health sensing despite using technology at the same rates as other racial groups.

OBJECTIVE

In this study, we aimed to understand the feasibility of continuous remote digital monitoring and ecological momentary assessments in Native Hawaiian, Filipino, and Pacific Islander communities in Hawaii by curating a novel data set of longitudinal Fitbit (Fitbit Inc) biosignals with the corresponding craving and substance use labels. We also aimed to develop personalized AI models that predict methamphetamine craving events in real time using wearable sensor data.

METHODS

We will develop personalized AI and machine learning models for methamphetamine use and craving prediction in 40 individuals from Native Hawaiian, Filipino, and Pacific Islander communities by curating a novel data set of real-time Fitbit biosensor readings and the corresponding participant annotations (ie, raw self-reported substance use data) of their methamphetamine use and cravings. In the process of collecting this data set, we will gain insights into cultural and other human factors that can challenge the proper acquisition of precise annotations. With the resulting data set, we will use self-supervised learning AI approaches, which are a new family of machine learning methods that allows a neural network to be trained without labels by being optimized to make predictions about the data. The inputs to the proposed AI models are Fitbit biosensor readings, and the outputs are predictions of methamphetamine use or craving. This paradigm is gaining increased attention in AI for health care.

RESULTS

To date, more than 40 individuals have expressed interest in participating in the study, and we have successfully recruited our first 5 participants with minimal logistical challenges and proper compliance. Several logistical challenges that the research team has encountered so far and the related implications are discussed.

CONCLUSIONS

We expect to develop models that significantly outperform traditional supervised methods by finetuning according to the data of a participant. Such methods will enable AI solutions that work with the limited data available from Native Hawaiian, Filipino, and Pacific Islander populations and that are inherently unbiased owing to their personalized nature. Such models can support future AI-powered digital therapeutics for substance abuse.

INTERNATIONAL REGISTERED REPORT IDENTIFIER (IRRID)

DERR1-10.2196/46493.

A comprehensive dataset for home appliance control using ERP-based BCIs with the application of inter-subject transfer learning

Brain-computer interfaces (BCIs) have a potential to revolutionize human-computer interaction by enabling direct links between the brain and computer systems. Recent studies are increasingly focusing on practical applications of BCIs-e.g., home appliance control just by thoughts. One of the non-invasive BCIs using electroencephalography (EEG) capitalizes on event-related potentials (ERPs) in response to target stimuli and have shown promise in controlling home appliance. In this paper, we present a comprehensive dataset of online ERP-based BCIs for controlling various home appliances in diverse stimulus presentation environments. We collected online BCI data from a total of 84 subjects among whom 60 subjects controlled three types of appliances (TV: 30, door lock: 15, and electric light: 15) with 4 functions per appliance, 14 subjects controlled a Bluetooth speaker with 6 functions via an LCD monitor, and 10 subjects controlled air conditioner with 4 functions via augmented reality (AR). Using the dataset, we aimed to address the issue of inter-subject variability in ERPs by employing the transfer learning in two different approaches. The first approach, "within-paradigm transfer learning," aimed to generalize the model within the same paradigm of stimulus presentation. The second approach, "cross-paradigm transfer learning," involved extending the model from a 4-class LCD environment to different paradigms. The results demonstrated that transfer learning can effectively enhance the generalizability of BCIs based on ERP across different subjects and environments.

An in-depth survey on Deep Learning-based Motor Imagery Electroencephalogram (EEG) classification

Electroencephalogram (EEG)-based Brain-Computer Interfaces (BCIs) build a communication path between human brain and external devices. Among EEG-based BCI paradigms, the most commonly used one is motor imagery (MI). As a hot research topic, MI EEG-based BCI has largely contributed to medical fields and smart home industry. However, because of the low signal-to-noise ratio (SNR) and the non-stationary characteristic of EEG data, it is difficult to correctly classify different types of MI-EEG signals. Recently, the advances in Deep Learning (DL) significantly facilitate the development of MI EEG-based BCIs. In this paper, we provide a systematic survey of DL-based MI-EEG classification methods. Specifically, we first comprehensively discuss several important aspects of DL-based MI-EEG classification, covering input formulations, network architectures, public datasets, etc. Then, we summarize problems in model performance comparison and give guidelines to future studies for fair performance comparison. Next, we fairly evaluate the representative DL-based models using source code released by the authors and meticulously analyse the evaluation results. By performing ablation study on the network architecture, we found that (1) effective feature fusion is indispensable for multi-stream CNN-based models. (2) LSTM should be combined with spatial feature extraction techniques to obtain good classification performance. (3) the use of dropout contributes little to improving the model performance, and that (4) adding fully connected layers to the models significantly increases their parameters but it might not improve their performance. Finally, we raise several open issues in MI-EEG classification and provide possible future research directions.

Implementation of artificial intelligence and machine learning-based methods in brain-computer interaction

Brain-computer interfaces are used for direct two-way communication between the human brain and the computer. Brain signals contain valuable information about the mental state and brain activity of the examined subject. However, due to their non-stationarity and susceptibility to various types of interference, their processing, analysis and interpretation are challenging. For these reasons, the research in the field of brain-computer interfaces is focused on the implementation of artificial intelligence, especially in five main areas: calibration, noise suppression, communication, mental condition estimation, and motor imagery. The use of algorithms based on artificial intelligence and machine learning has proven to be very promising in these application domains, especially due to their ability to predict and learn from previous experience. Therefore, their implementation within medical technologies can contribute to more accurate information about the mental state of subjects, alleviate the consequences of serious diseases or improve the quality of life of disabled patients.

Personalized Machine Learning using Passive Sensing and Ecological Momentary Assessments for Meth Users in Hawaii: A Research Protocol

Background

Artificial intelligence (AI)-powered digital therapies which detect meth cravings delivered on consumer devices have the potential to reduce these disparities by providing remote and accessible care solutions to Native Hawaiians, Filipinos, and Pacific Islanders (NHFPI) communities with limited care solutions. However, NHFPI are fully understudied with respect to digital therapeutics and AI health sensing despite using technology at the same rates as other races.

Objective

We seek to fulfill two research aims: (1) Understand the feasibility of continuous remote digital monitoring and ecological momentary assessments (EMAs) in NHFPI in Hawaii by curating a novel dataset of longitudinal FitBit biosignals with corresponding craving and substance use labels. (2) Develop personalized AI models which predict meth craving events in real time using wearable sensor data.

Methods

We will develop personalized AI/ML (artificial intelligence/machine learning) models for meth use and craving prediction in 40 NHFPI individuals by curating a novel dataset of real-time FitBit biosensor readings and corresponding participant annotations (i.e., raw self-reported substance use data) of their meth use and cravings. In the process of collecting this dataset, we will glean insights about cultural and other human factors which can challenge the proper acquisition of precise annotations. With the resulting dataset, we will employ self-supervised learning (SSL) AI approaches, which are a new family of ML methods that allow a neural network to be trained without labels by being optimized to make predictions about the data itself. The inputs to the proposed AI models are FitBit biosensor readings and the outputs are predictions of meth use or craving. This paradigm is gaining increased attention in AI for healthcare.

Conclusions

We expect to develop models which significantly outperform traditional supervised methods by fine-tuning to an individual subject's data. Such methods will enable AI solutions which work with the limited data available from NHFPI populations and which are inherently unbiased due to their personalized nature. Such models can support future AI-powered digital therapeutics for substance abuse.

Hand-worn devices for assessment and rehabilitation of motor function and their potential use in BCI protocols: a review

Introduction

Various neurological conditions can impair hand function. Affected individuals cannot fully participate in activities of daily living due to the lack of fine motor control. Neurorehabilitation emphasizes repetitive movement and subjective clinical assessments that require clinical experience to administer.

Methods

Here, we perform a review of literature focused on the use of hand-worn devices for rehabilitation and assessment of hand function. We paid particular attention to protocols that involve brain-computer interfaces (BCIs) since BCIs are gaining ground as a means for detecting volitional signals as the basis for interactive motor training protocols to augment recovery. All devices reviewed either monitor, assist, stimulate, or support hand and finger movement.

Results

A majority of studies reviewed here test or validate devices through clinical trials, especially for stroke. Even though sensor gloves are the most commonly employed type of device in this domain, they have certain limitations. Many such gloves use bend or inertial sensors to monitor the movement of individual digits, but few monitor both movement and applied pressure. The use of such devices in BCI protocols is also uncommon.

Discussion

We conclude that hand-worn devices that monitor both flexion and grip will benefit both clinical diagnostic assessment of function during treatment and closed-loop BCI protocols aimed at rehabilitation.

State-of-the-Art on Brain-Computer Interface Technology

This paper provides a comprehensive overview of the state-of-the-art in brain-computer interfaces (BCI). It begins by providing an introduction to BCIs, describing their main operation principles and most widely used platforms. The paper then examines the various components of a BCI system, such as hardware, software, and signal processing algorithms. Finally, it looks at current trends in research related to BCI use for medical, educational, and other purposes, as well as potential future applications of this technology. The paper concludes by highlighting some key challenges that still need to be addressed before widespread adoption can occur. By presenting an up-to-date assessment of the state-of-the-art in BCI technology, this paper will provide valuable insight into where this field is heading in terms of progress and innovation.

Machine-Learning Methods for Speech and Handwriting Detection Using Neural Signals: A Review

Brain-Computer Interfaces (BCIs) have become increasingly popular in recent years due to their potential applications in diverse fields, ranging from the medical sector (people with motor and/or communication disabilities), cognitive training, gaming, and Augmented Reality/Virtual Reality (AR/VR), among other areas. BCI which can decode and recognize neural signals involved in speech and handwriting has the potential to greatly assist individuals with severe motor impairments in their communication and interaction needs. Innovative and cutting-edge advancements in this field have the potential to develop a highly accessible and interactive communication platform for these people. The purpose of this review paper is to analyze the existing research on handwriting and speech recognition from neural signals. So that the new researchers who are interested in this field can gain thorough knowledge in this research area. The current research on neural signal-based recognition of handwriting and speech has been categorized into two main types: invasive and non-invasive studies. We have examined the latest papers on converting speech-activity-based neural signals and handwriting-activity-based neural signals into text data. The methods of extracting data from the brain have also been discussed in this review. Additionally, this review includes a brief summary of the datasets, preprocessing techniques, and methods used in these studies, which were published between 2014 and 2022. This review aims to provide a comprehensive summary of the methodologies used in the current literature on neural signal-based recognition of handwriting and speech. In essence, this article is intended to serve as a valuable resource for future researchers who wish to investigate neural signal-based machine-learning methods in their work.

Advanced Modeling and Signal Processing Methods in Brain-Computer Interfaces Based on a Vector of Cyclic Rhythmically Connected Random Processes

In this study is substantiated the new mathematical model of vector of electroencephalographic signals, registered under the conditions of multiple repetitions of the mental control influences of brain-computer interface operator, in the form of a vector of cyclic rhythmically connected random processes, which, due to taking into account the stochasticity and cyclicity, the variability and commonality of the rhythm of the investigated signals have a number of advantages over the known models. This new model opens the way for the study of multidimensional distribution functions; initial, central, and mixed moment functions of higher order such as for each electroencephalographic signal separately; as well as for their respective compatible probabilistic characteristics, among which the most informative characteristics can be selected. This provides an increase in accuracy in the detection (classification) of mental control influences of the brain-computer interface operators. Based on the developed mathematical model, the statistical processing methods of vector of electroencephalographic signals are substantiated, which consist of statistical evaluation of its probabilistic characteristics and make it possible to conduct an effective joint statistical estimation of the probability characteristics of electroencephalographic signals. This provides the basis for coordinated integration of information from different sensors. The use of moment functions of higher order and their spectral images in the frequency domain, as informative characteristics in brain-computer interface systems, are substantiated. Their significant sensitivity to the mental controlling influence of the brain-computer interface operator is experimentally established. The application of Bessel's inequality to the problems of reducing the dimensions (from 500 to 20 numbers) of the vectors of informative features makes it possible to significantly reduce the computational complexity of the algorithms for the functioning of brain-computer interface systems. Namely, we experimentally established that only the first 20 values of the Fourier transform of the estimation of moment functions of higher-order electroencephalographic signals are sufficient to form the vector of informative features in brain-computer interface systems, because these spectral components make up at least 95% of the total energy of the corresponding statistical estimate of the moment functions of higher-order electroencephalographic signals.

Identifying Thematics in a Brain-Computer Interface Research

This umbrella review is motivated to understand the shift in research themes on brain-computer interfacing (BCI) and it determined that a shift away from themes that focus on medical advancement and system development to applications that included education, marketing, gaming, safety, and security has occurred. The background of this review examined aspects of BCI categorisation, neuroimaging methods, brain control signal classification, applications, and ethics. The specific area of BCI software and hardware development was not examined. A search using One Search was undertaken and 92 BCI reviews were selected for inclusion. Publication demographics indicate the average number of authors on review papers considered was 4.2 ± 1.8. The results also indicate a rapid increase in the number of BCI reviews from 2003, with only three reviews before that period, two in 1972, and one in 1996. While BCI authors were predominantly Euro-American in early reviews, this shifted to a more global authorship, which China dominated by 2020-2022. The review revealed six disciplines associated with BCI systems: life sciences and biomedicine (n = 42), neurosciences and neurology (n = 35), and rehabilitation (n = 20); (2) the second domain centred on the theme of functionality: computer science (n = 20), engineering (n = 28) and technology (n = 38). There was a thematic shift from understanding brain function and modes of interfacing BCI systems to more applied research novel areas of research-identified surround artificial intelligence, including machine learning, pre-processing, and deep learning. As BCI systems become more invasive in the lives of "normal" individuals, it is expected that there will be a refocus and thematic shift towards increased research into ethical issues and the need for legal oversight in BCI application.

On closed-loop brain stimulation systems for improving the quality of life of patients with neurological disorders

Emerging brain technologies have significantly transformed human life in recent decades. For instance, the closed-loop brain-computer interface (BCI) is an advanced software-hardware system that interprets electrical signals from neurons, allowing communication with and control of the environment. The system then transmits these signals as controlled commands and provides feedback to the brain to execute specific tasks. This paper analyzes and presents the latest research on closed-loop BCI that utilizes electric/magnetic stimulation, optogenetic, and sonogenetic techniques. These techniques have demonstrated great potential in improving the quality of life for patients suffering from neurodegenerative or psychiatric diseases. We provide a comprehensive and systematic review of research on the modalities of closed-loop BCI in recent decades. To achieve this, the authors used a set of defined criteria to shortlist studies from well-known research databases into categories of brain stimulation techniques. These categories include deep brain stimulation, transcranial magnetic stimulation, transcranial direct-current stimulation, transcranial alternating-current stimulation, and optogenetics. These techniques have been useful in treating a wide range of disorders, such as Alzheimer's and Parkinson's disease, dementia, and depression. In total, 76 studies were shortlisted and analyzed to illustrate how closed-loop BCI can considerably improve, enhance, and restore specific brain functions. The analysis revealed that literature in the area has not adequately covered closed-loop BCI in the context of cognitive neural prosthetics and implanted neural devices. However, the authors demonstrate that the applications of closed-loop BCI are highly beneficial, and the technology is continually evolving to improve the lives of individuals with various ailments, including those with sensory-motor issues or cognitive deficiencies. By utilizing emerging techniques of stimulation, closed-loop BCI can safely improve patients' cognitive and affective skills, resulting in better healthcare outcomes.

Event-Related Potential-Based Brain-Computer Interface Using the Thai Vowels' and Numerals' Auditory Stimulus Pattern

Herein, we developed an auditory stimulus pattern for an event-related potential (ERP)-based brain-computer interface (BCI) system to improve control and communication in quadriplegia with visual impairment. Auditory stimulus paradigms for multicommand electroencephalogram (EEG)-based BCIs and audio stimulus patterns were examined. With the proposed auditory stimulation, using the selected Thai vowel, similar to the English vowel, and Thai numeral sounds, as simple target recognition, we explored the ERPs' response and classification efficiency from the suggested EEG channels. We also investigated the use of single and multi-loudspeakers for auditory stimuli. Four commands were created using the proposed paradigm. The experimental paradigm was designed to observe ERP responses and verify the proposed auditory stimulus pattern. The conventional classification method produced four commands using the proposed auditory stimulus pattern. The results established that the proposed auditory stimulation with 20 to 30 trials of stream stimuli could produce a prominent ERP response from Pz channels. The vowel stimuli could achieve higher accuracy than the proposed numeral stimuli for two auditory stimuli intervals (100 and 250 ms). Additionally, multi-loudspeaker patterns through vowel and numeral sound stimulation provided an accuracy greater than 85% of the average accuracy. Thus, the proposed auditory stimulation patterns can be implemented as a real-time BCI system to aid in the daily activities of quadratic patients with visual and tactile impairments. In future, practical use of the auditory ERP-based BCI system will be demonstrated and verified in an actual scenario.

Neurosurgical Team Acceptability of Brain-Computer Interfaces: A Two-Stage International Cross-Sectional Survey

OBJECTIVE

Invasive brain-computer interfaces (BCIs) require neurosurgical implantation, which confers a range of risks. Despite this situation, no studies have assessed the acceptability of invasive BCIs among the neurosurgical team. This study aims to establish baseline knowledge of BCIs within the neurosurgical team and identify attitudes toward different applications of invasive BCI.

METHODS

A 2-stage cross-sectional international survey of the neurosurgical team (neurosurgeons, anesthetists, and operating room nurses) was conducted. Results from the first, qualitative, survey were used to guide the second-stage quantitative survey, which assessed acceptability of invasive BCI applications. Five-part Likert scales were used to collect quantitative data. Surveys were distributed internationally via social media and collaborators.

RESULTS

A total of 108 qualitative responses were collected. Themes included the promise of BCIs positively affecting disease targets, concerns regarding stability, and an overall positive emotional reaction to BCI technology. The quantitative survey generated 538 responses from 32 countries. Baseline knowledge of BCI technology was poor, with 9% claiming to have a good or expert knowledge of BCIs. Acceptability of invasive BCI for rehabilitative purposes was >80%. Invasive BCI for augmentation in healthy populations divided opinion.

CONCLUSIONS

The neurosurgical team's view of the acceptability of invasive BCI was divided across a range of indications. Some applications (e.g., stroke rehabilitation) were viewed as more appropriate than other applications (e.g., augmentation for military use). This range in views highlights the need for stakeholder consultation on acceptable use cases along with regulation and guidance to govern initial BCI implantations if patients are to realize the potential benefits.

Factors influencing students' adoption intention of brain–computer interfaces in a game-learning context

Purpose

A research framework that explains adoption intention in students with regard to brain–computer interface (BCI) games in the learning context was proposed and empirically examined.

Design/methodology/approach

In this study, an approach integrating the decomposed theory of planned behavior, perceived playfulness, risk and the task–technology fit (TTF) concept was used to assess data collected using a post-experiment questionnaire from a student sample in Taiwan. The research model was tested using the partial least-squares structural equation modeling (PLS-SEM) technique.

Findings

Attitude, subjective norms and TTF were shown to impact intention to play the BCI game significantly, while perceived behavioral control did not show a significant impact. The influence of superiors and peers was found to positively predict subjective norms. With the exception of perceived ease of use, all of the proposed antecedents were found to impact attitude toward BCI games. Technology facilitating conditions and BCI technology characteristics were shown to positively determine perceived behavior control and TTF, respectively. However, the other proposed factors did not significantly influence the latter two dependents.

Originality/value

This research contributes to the nascent literature on BCI games in the context of learning by highlighting the influence of belief-related psychological factors on user acceptance of BCI games. Moreover, this study highlights the important, respective influences of perceived playfulness, risk and TTF on users' perceptions of a game, body monitoring and technology implementation, each of which is known to influence willingness to play.

Deep Residual Convolutional Neural Networks for Brain-Computer Interface to Visualize Neural Processing of Hand Movements in the Human Brain

Concomitant with the development of deep learning, brain-computer interface (BCI) decoding technology has been rapidly evolving. Convolutional neural networks (CNNs), which are generally used as electroencephalography (EEG) classification models, are often deployed in BCI prototypes to improve the estimation accuracy of a participant's brain activity. However, because most BCI models are trained, validated, and tested via within-subject cross-validation and there is no corresponding generalization model, their applicability to unknown participants is not guaranteed. In this study, to facilitate the generalization of BCI model performance to unknown participants, we trained a model comprising multiple layers of residual CNNs and visualized the reasons for BCI classification to reveal the location and timing of neural activities that contribute to classification. Specifically, to develop a BCI that can distinguish between rest, left-hand movement, and right-hand movement tasks with high accuracy, we created multilayers of CNNs, inserted residual networks into the multilayers, and used a larger dataset than in previous studies. The constructed model was analyzed with gradient-class activation mapping (Grad-CAM). We evaluated the developed model via subject cross-validation and found that it achieved significantly improved accuracy (85.69 ± 1.10%) compared with conventional models or without residual networks. Grad-CAM analysis of the classification of cases in which our model produced correct answers showed localized activity near the premotor cortex. These results confirm the effectiveness of inserting residual networks into CNNs for tuning BCI. Further, they suggest that recording EEG signals over the premotor cortex and some other areas contributes to high classification accuracy.

Spike-Representation of EEG Signals for Performance Enhancement of Brain-Computer Interfaces

Brain-computer interfaces (BCI) relying on electroencephalography (EEG) based neuroimaging mode has shown prospects for real-world usage due to its portability and optional selectivity of fewer channels for compactness. However, noise and artifacts often limit the capacity of BCI systems especially for event-related potentials such as P300 and error-related negativity (ERN), whose biomarkers are present in short time segments at the time-series level. Contrary to EEG, invasive recording is less prone to noise but requires a tedious surgical procedure. But EEG signal is the result of aggregation of neuronal spiking information underneath the scalp surface and transforming the relevant BCI task's EEG signal to spike representation could potentially help improve the BCI performance. In this study, we designed an approach using a spiking neural network (SNN) which is trained using surrogate-gradient descent to generate task-related multi-channel EEG template signals of all classes. The trained model is in turn leveraged to obtain the latent spike representation for each EEG sample. Comparing the classification performance of EEG signal and its spike-representation, the proposed approach enhanced the performance of ERN dataset from 79.22 to 82.27% with naive bayes and for P300 dataset, the accuracy was improved from 67.73 to 69.87% using xGboost. In addition, principal component analysis and correlation metrics were evaluated on both EEG signals and their spike-representation to identify the reason for such improvement.

Toward EEG-Based BCI Applications for Industry 4.0: Challenges and Possible Applications

In the last few decades, Brain-Computer Interface (BCI) research has focused predominantly on clinical applications, notably to enable severely disabled people to interact with the environment. However, recent studies rely mostly on the use of non-invasive electroencephalographic (EEG) devices, suggesting that BCI might be ready to be used outside laboratories. In particular, Industry 4.0 is a rapidly evolving sector that aims to restructure traditional methods by deploying digital tools and cyber-physical systems. BCI-based solutions are attracting increasing attention in this field to support industrial performance by optimizing the cognitive load of industrial operators, facilitating human-robot interactions, and make operations in critical conditions more secure. Although these advancements seem promising, numerous aspects must be considered before developing any operational solutions. Indeed, the development of novel applications outside optimal laboratory conditions raises many challenges. In the current study, we carried out a detailed literature review to investigate the main challenges and present criteria relevant to the future deployment of BCI applications for Industry 4.0.

An Open Source-Based BCI Application for Virtual World Tour and Its Usability Evaluation

Brain-computer interfaces can provide a new communication channel and control functions to people with restricted movements. Recent studies have indicated the effectiveness of brain-computer interface (BCI) applications. Various types of applications have been introduced so far in this field, but the number of those available to the public is still insufficient. Thus, there is a need to expand the usability and accessibility of BCI applications. In this study, we introduce a BCI application for users to experience a virtual world tour. This software was built on three open-source environments and is publicly available through the GitHub repository. For a usability test, 10 healthy subjects participated in an electroencephalography (EEG) experiment and evaluated the system through a questionnaire. As a result, all the participants successfully played the BCI application with 96.6% accuracy with 20 blinks from two sessions and gave opinions on its usability (e.g., controllability, completeness, comfort, and enjoyment) through the questionnaire. We believe that this open-source BCI world tour system can be used in both research and entertainment settings and hopefully contribute to open science in the BCI field.

A Comparison of Conventional and Tri-Polar EEG Electrodes for Decoding Real and Imaginary Finger Movements from One Hand

The representations of different fingers in the sensorimotor cortex are largely overlapped, which necessitate a good signal-to-noise ratio (SNR) and high spatial resolution to classify individual finger movements from one hand. Electroencephalography (EEG) recorded with disc electrodes has low SNR and poor spatial resolution. The surface Laplacian has been applied to EEG to improve the spatial resolution and selectivity of the surface electrical activity recording. Tri-polar concentric ring electrodes (TCREs) were shown to estimate the Laplacian automatically with better spatial resolution than disc electrodes. For this work, movement-related potentials (MRPs) were recorded from four TCREs and disc electrodes while 13 subjects performed real and imaginary finger movements. The MRP signals recorded with the TCREs have significantly less mutual information and coherence between neighboring locations compared to disc electrodes. The results also show that signals from TCREs generated higher accuracy compared to disc electrodes. It further shows that TCREs using temporal EEG data as features yield an average accuracy of [Formula: see text]% and [Formula: see text]% for real and imaginary finger movements, respectively, which is significantly higher than utilizing EEG spectral power changes in [Formula: see text] and [Formula: see text] bands as features. Similarly, with the disc electrodes, it achieved highest accuracy of [Formula: see text]% and [Formula: see text]% for real and imaginary finger movements, respectively, with temporal EEG data feature.

Emotion-Driven Analysis and Control of Human-Robot Interactions in Collaborative Applications

The utilization of robotic systems has been increasing in the last decade. This increase has been derived by the evolvement in the computational capabilities, communication systems, and the information systems of the manufacturing systems which is reflected in the concept of Industry 4.0. Furthermore, the robotics systems are continuously required to address new challenges in the industrial and manufacturing domain, like keeping humans in the loop, among other challenges. Briefly, the keeping humans in the loop concept focuses on closing the gap between humans and machines by introducing a safe and trustworthy environment for the human workers to work side by side with robots and machines. It aims at increasing the engagement of the human as the automation level increases rather than replacing the human, which can be nearly impossible in some applications. Consequently, the collaborative robots (Cobots) have been created to allow physical interaction with the human worker. However, these cobots still lack of recognizing the human emotional state. In this regard, this paper presents an approach for adapting cobot parameters to the emotional state of the human worker. The approach utilizes the Electroencephalography (EEG) technology for digitizing and understanding the human emotional state. Afterwards, the parameters of the cobot are instantly adjusted to keep the human emotional state in a desirable range which increases the confidence and the trust between the human and the cobot. In addition, the paper includes a review on technologies and methods for emotional sensing and recognition. Finally, this approach is tested on an ABB YuMi cobot with commercially available EEG headset.

Applications of Biological and Physiological Signals in Commercial Video Gaming and Game Research: A Review

Video gaming is now available as a fully immersive experience that creates responsive inputs and outputs concerning the user, and some experimental developers have integrated the use of the voice, brain, or muscles as input controls. The use of physiological signal equipment can provide valuable information regarding the emotion of a player or patient during gameplay. In this article, we discuss five of the most common biosignals that are used in gaming research, and their function and devices that may be used for measurement. We break down those individual signals and present examples of research studies that implement them. We also discuss the usage of biological signals within commercial gaming and conclude with some possible future directions for the use of biological signals in gaming and game research.

SSVEP-Based Brain-Computer Interface With a Limited Number of Frequencies Based on Dual-Frequency Biased Coding

How to encode as many targets as possible with a limited-frequency resource is a difficult problem in the practical use of a steady-state visual evoked potential (SSVEP) based brain-computer interface (BCI) speller. To solve this problem, this study developed a novel method called dual-frequency biased coding (DFBC) to tag targets in a SSVEP-based 48-character virtual speller, in which each target is encoded with a permutation sequence consisting of two permuted flickering periods that flash at different frequencies. The proposed paradigm was validated by 11 participants in an offline experiment and 7 participants in an online experiment. Three occipital channels (O1, Oz, and O2) were used to obtain the SSVEP signals for identifying the targets. Based on the coding characteristics of the DFBC method, the proposed approach has the ability of self-correction and thus achieves an accuracy of 76.6% and 79.3% for offline and online experiments, respectively, which outperforms the traditional multiple frequencies sequential coding (MFSC) method. This study demonstrates that DFBC is an efficient method for coding a high number of SSVEP targets with a small number of available frequencies.

Gamma frequencies as a predictor for the accomplishment of a motor task guided through the action observation network

BACKGROUND

Action observation describes a concept where the subsequent motor behavior of an individual can be modulated though observing an action. This occurs through the activation of neurons in the action observation network, acting on a variety of motor learning processes. This network has been proven highly useful in the rehabilitation of patients with acquired brain injury, placing "action observation" as one of the most effective techniques for motor recovery in physical neurorehabilitation.

OBJECTIVE

The aim of this paper is to define an EEG marker for motor learning, guided through observation.

METHODS

Healthy subjects (n = 41) participated voluntarily for this research. They were asked to repeat an unknown motor behavior, immediately after observing a video. During the observation, EEG raw signals where collected with a portable EEG and the results were later compared with success and fail on repeating the motor procedure. The comparison was then analyzed with the Mann-Whitney U test for non-parametrical data, with a confidence interval of 95%.

RESULTS

A significant relation between motor performance and neural activity was found for Alpha (p = 0,0149) and Gamma (0,0005) oscillatory patterns.

CONCLUSION

Gamma oscillations with frequencies between 41 and 49,75 Hz, seem to be an adequate EEG marker for motor performance guided through the action observation network. The technology used for this paper is easy to use, low-cost and presents valid measurements for the recommended oscillatory frequencies, implying a possible use on rehabilitation, by collecting data in real-time during therapeutic interventions and assessments.

Designing a flexible tool for rapid implementation of brain-computer interfaces (BCI) in game development

Early neurological injury or disease can lead to severe life-long physical impairments, despite normal cognitive function. For such individuals, brain-computer interfaces (BCI) may provide a means to regain access to the world by offering control of systems through directly processing brain patterns. However, current BCI applications are often research driven and consequently seen as uninteresting, particularly for prolonged use and younger BCI-users. To help mitigate this concern, this paper establishes a tool for researchers and game developers alike to rapidly incorporate a BCI control scheme (the P300 oddball response) into a gaming environment. Preliminary results indicate the proposed P300 Dynamic Cube (PDC) asset works in online BCI environments (n=20, healthy adult participants), resulting in median classification accuracy of 75 ± 3.28%. Additionally, the PDC tool can be rapidly adapted for a variety of game designs, evidenced by its incorporation into submissions to the Brain-Computer Interface (BCI) Game Jam 2019 competition. These findings support the PDC as a useful asset in the design and development of BCI-based games.

Brain-Computer Interface-Based Humanoid Control: A Review

A Brain-Computer Interface (BCI) acts as a communication mechanism using brain signals to control external devices. The generation of such signals is sometimes independent of the nervous system, such as in Passive BCI. This is majorly beneficial for those who have severe motor disabilities. Traditional BCI systems have been dependent only on brain signals recorded using Electroencephalography (EEG) and have used a rule-based translation algorithm to generate control commands. However, the recent use of multi-sensor data fusion and machine learning-based translation algorithms has improved the accuracy of such systems. This paper discusses various BCI applications such as tele-presence, grasping of objects, navigation, etc. that use multi-sensor fusion and machine learning to control a humanoid robot to perform a desired task. The paper also includes a review of the methods and system design used in the discussed applications.

Current Status, Challenges, and Possible Solutions of EEG-Based Brain-Computer Interface: A Comprehensive Review

Brain-Computer Interface (BCI), in essence, aims at controlling different assistive devices through the utilization of brain waves. It is worth noting that the application of BCI is not limited to medical applications, and hence, the research in this field has gained due attention. Moreover, the significant number of related publications over the past two decades further indicates the consistent improvements and breakthroughs that have been made in this particular field. Nonetheless, it is also worth mentioning that with these improvements, new challenges are constantly discovered. This article provides a comprehensive review of the state-of-the-art of a complete BCI system. First, a brief overview of electroencephalogram (EEG)-based BCI systems is given. Secondly, a considerable number of popular BCI applications are reviewed in terms of electrophysiological control signals, feature extraction, classification algorithms, and performance evaluation metrics. Finally, the challenges to the recent BCI systems are discussed, and possible solutions to mitigate the issues are recommended.

A Systematic Review of Closed-Loop Feedback Techniques in Sleep Studies-Related Issues and Future Directions

Advances in computer processing technology have enabled researchers to analyze real-time brain activity and build real-time closed-loop paradigms. In many fields, the effectiveness of these closed-loop protocols has proven to be better than that of the simple open-loop paradigms. Recently, sleep studies have attracted much attention as one possible application of closed-loop paradigms. To date, several studies that used closed-loop paradigms have been reported in the sleep-related literature and recommend a closed-loop feedback system to enhance specific brain activity during sleep, which leads to improvements in sleep's effects, such as memory consolidation. However, to the best of our knowledge, no report has reviewed and discussed the detailed technical issues that arise in designing sleep closed-loop paradigms. In this paper, we reviewed the most recent reports on sleep closed-loop paradigms and offered an in-depth discussion of some of their technical issues. We found 148 journal articles strongly related with 'sleep and stimulation' and reviewed 20 articles on closed-loop feedback sleep studies. We focused on human sleep studies conducting any modality of feedback stimulation. Then we introduced the main component of the closed-loop system and summarized several open-source libraries, which are widely used in closed-loop systems, with step-by-step guidelines for closed-loop system implementation for sleep. Further, we proposed future directions for sleep research with closed-loop feedback systems, which provide some insight into closed-loop feedback systems.

Emotion-Inducing Imagery Versus Motor Imagery for a Brain-Computer Interface

Neural correlates of intentionally induced human emotions may offer alternative imagery strategies to control brain-computer interface (BCI) applications. In this paper, a novel BCI control strategy i.e., imagining fictional or recalling mnemonic sad and happy events, emotion-inducing imagery (EII), is compared to motor imagery (MI) in a study involving multiple sessions using a two-class electroencephalogram (EEG)-based BCI paradigm with 12 participants. The BCI setup enabled online continuous visual feedback presentation in a game involving one-dimensional control of a game character. MI and EII are compared across different signal-processing frameworks which are based on neural-time-series-prediction-preprocessing (NTSPP), filter bank common spatial patterns (FBCSP) and hemispheric asymmetry (ASYM). Online single-trial classification accuracies (CA) results indicate that MI performance across all participants is 77.54% compared to EII performance of 68.78% ( ). The results show that an ensemble of the NTSPP, FBCSP and ASYM frameworks maximizes performance for EII with average CA of 71.64% across all participants. Furthermore, the participants' subjective responses indicate that they preferred MI over emotion-inducing imagery (EII) in controlling the game character, and MI was perceived to offer most control over the game character. The results suggest that EII is not a viable alternative to MI for the majority of participants in this study but may be an alternative imagery for a subset of BCI users based on acceptable EII performance (CA >70%) observed for some participants.

Two Player Online Brain-Controlled Chess

Brain-computer interfaces (BCIs) allow for translating brain signals into control commands, e.g. to control games. One of the biggest quests of the BCI community is to realize new exciting applications. In this paper, we present a two player online chess application where both players control their pieces solely with their brain activity. Control is realized in a two-step process where players first select the desired chess piece and then the field they want to move it to. A selection is accomplished with visual event-related potentials that are elicited by highlighting each possible piece or field one by one. Six healthy volunteers participated in our study by playing a game against each other in pairs over a free chess server. They successfully completed at least one match per pair. Our results show that even BCI-naive players obtain almost perfect control over the application. On average, 96% of the moves were correct. Most of the errors came from technical problems that can be corrected in future versions of the application. Given the high popularity of chess, this intuitive BCI game is an appealing new application for patients and for healthy users that want to explore BCIs.

Brain-computer interfaces and virtual reality for neurorehabilitation

Brain-computer interfaces (BCIs) and virtual reality (VR) are two technologic advances that are changing our way of interacting with the world. BCIs can be used to influence and can serve as a control mechanism in navigation tasks, communication, or other assistive functions. VR can create ad hoc interactive scenarios that involve all our senses, stimulate the brain in a multisensory fashion, and increase the motivation and fun with game-like environments. VR and motion tracking enable natural human-computer interaction at cognitive and physical levels. This includes both brain and body in the design of meaningful VR experiences; these cases in which participants feel naturally present could help augment the benefits of BCIs for assistive and neurorehabilitation applications for the relearning of motor and cognitive skills. VR technology is now available at the consumer level thanks to the proliferation of affordable head-mounted displays (HMDs). Merging both technologies into simplified, practical devices may help democratize these technologies.

Online Home Appliance Control Using EEG-Based Brain–Computer Interfaces

Brain–computer interfaces (BCIs) allow patients with paralysis to control external devices by mental commands. Recent advances in home automation and the Internet of things may extend the horizon of BCI applications into daily living environments at home. In this study, we developed an online BCI based on scalp electroencephalography (EEG) to control home appliances. The BCI users controlled TV channels, a digital door-lock system, and an electric light system in an unshielded environment. The BCI was designed to harness P300 and N200 components of event-related potentials (ERPs). On average, the BCI users could control TV channels with an accuracy of 83.0% ± 17.9%, the digital door-lock with 78.7% ± 16.2% accuracy, and the light with 80.0% ± 15.6% accuracy, respectively. Our study demonstrates a feasibility to control multiple home appliances using EEG-based BCIs.

A new algorithm for blink correction adaptive to inter- and intra-subject variability

Electroencephalographic (EEG) signals are constantly superimposed with biological artifacts. In particular, spontaneous blinks represent a recurrent event that cannot be easily avoided. The main goal of this paper is to present a new algorithm for blink correction (ABC) that is adaptive to inter- and intra-subject variability. The whole process of designing a Brain-Computer Interface (BCI)-based EEG experiment is highlighted. From sample size determination to classification, a mixture of the standardized low-resolution electromagnetic tomography (sLORETA) for source localization and time restriction, followed by Riemannian geometry classifiers is featured. Comparison between ABC and the commonly-used Independent Component Analysis (ICA) for blinks removal shows a net amelioration with ABC. With the same pipeline using uncorrected data as a reference, ABC improves classification by 5.38% in average, whereas ICA deteriorates by -2.67%. Furthermore, while ABC accurately reconstructs blink-free data from simulated data, ICA yields a potential difference up to 200% from the original blink-free signal and an increased variance of 30.42%. Finally, ABC's major advantages are ease of visualization and understanding, low computation load favoring simple real-time implementation, and lack of spatial filtering, which allows for more flexibility during the classification step.

Effects of Task Complexity on Motor Imagery-Based Brain-Computer Interface

The performance of electroencephalogram (EEG)-based brain-computer interfaces (BCIs) still needs improvements for real world applications. An improvement on BCIs could be achieved by enhancing brain signals from the source via subject intention-based modulation. In this work, we aim to investigate the effects of task complexity on performance of motor imagery (MI) based BCIs. In specific, we studied the effects of motor imagery of a complex task versus a simple task on discriminability of brain activation patterns using EEG. The results show an increase of up to 7.25% in BCI classification accuracy for motor imagery of the complex task in comparison to the simple task. Furthermore, spectral power analysis in low frequency bands, alpha and beta, shows a significant decrease in power value for the complex task. However, high frequency gamma band analysis unveils a significant increase for the complex task. These findings may lead to designing better BCIs with high performance.

Common spatial pattern and wavelet decomposition for motor imagery EEG- fTCD brain-computer interface

BACKGROUND

Recently, hybrid brain-computer interfaces (BCIs) combining more than one modality have been investigated with the aim of boosting the performance of the existing single-modal BCIs in terms of accuracy and information transfer rate (ITR). Previously, we introduced a novel hybrid BCI in which EEG and fTCD modalities are used simultaneously to measure electrical brain activity and cerebral blood velocity during motor imagery (MI) tasks.

NEW METHOD

In this paper, we used multi-scale analysis and common spatial pattern algorithm to extract EEG and fTCD features. Moreover, we proposed probabilistic fusion of EEG and fTCD evidences instead of concatenating EEG and fTCD feature vectors corresponding to each trial. A Bayesian approach was proposed to fuse EEG and fTCD evidences under 3 different assumptions.

RESULTS

Experimental results showed that 93.85%, 93.71%, and 100% average accuracies and 19.89, 26.55, and 40.83 bits/min average ITRs were achieved for right MI vs baseline, left MI versus baseline, and right MI versus left MI respectively.

COMPARISON WITH EXISTING METHODS

These performance measures outperformed the results we obtained before in our preliminary study in which average accuracies of 88.33%, 89.48%, and 82.38% and average ITRs of 4.17, 5.45, and 10.57 bits/min were achieved for right MI versus baseline, left MI versus baseline, and right MI versus left MI respectively. Moreover, in terms of both accuracy and speed, the EEG- fTCD BCI with the proposed analysis techniques outperformed all EEG- fNIRS studies in comparison.

CONCLUSIONS

The proposed system is a more accurate and faster alternative to EEG-fNIRS systems.

Using brain-computer interfaces: a scoping review of studies employing social research methods

Background

The rapid expansion of research on Brain-Computer Interfaces (BCIs) is not only due to the promising solutions offered for persons with physical impairments. There is also a heightened need for understanding BCIs due to the challenges regarding ethics presented by new technology, especially in its impact on the relationship between man and machine. Here we endeavor to present a scoping review of current studies in the field to gain insight into the complexity of BCI use. By examining studies related to BCIs that employ social research methods, we seek to demonstrate the multitude of approaches and concerns from various angles in considering the social and human impact of BCI technology.

Methods

For this scoping review of research on BCIs’ social and ethical implications, we systematically analyzed six databases, encompassing the fields of medicine, psychology, and the social sciences, in order to identify empirical studies on BCIs. The search yielded 73 publications that employ quantitative, qualitative, or mixed methods.

Results

Of the 73 publications, 71 studies address the user perspective. Some studies extend to consideration of other BCI stakeholders such as medical technology experts, caregivers, or health care professionals. The majority of the studies employ quantitative methods. Recurring themes across the studies examined were general user opinion towards BCI, central technical or social issues reported, requests/demands made by users of the technology, the potential/future of BCIs, and ethical aspects of BCIs.

Conclusions

Our findings indicate that while technical aspects of BCIs such as usability or feasibility are being studied extensively, comparatively little in-depth research has been done on the self-image and self-experience of the BCI user. In general there is also a lack of focus or examination of the caregiver’s perspective.

Electronic supplementary material

The online version of this article (10.1186/s12910-019-0354-1) contains supplementary material, which is available to authorized users.