Brain Painting: First Evaluation of a New Brain-Computer Interface Application with ALS-Patients and Healthy Volunteers

Diffusion model-based image generation from rat brain activity

Brain-computer interface (BCI) technology has gained recognition in various fields, including clinical applications, assistive technology, and human-computer interaction research. BCI enables communication, control, and monitoring of the affective/cognitive states of users. Recently, BCI has also found applications in the artistic field, enabling real-time art composition using brain activity signals, and engaging performers, spectators, or an entire audience with brain activity-based artistic environments. Existing techniques use specific features of brain activity, such as the P300 wave and SSVEPs, to control drawing tools, rather than directly reflecting brain activity in the output image. In this study, we present a novel approach that uses a latent diffusion model, a type of deep neural network, to generate images directly from continuous brain activity. We demonstrate this technology using local field potentials from the neocortex of freely moving rats. This system continuously converted the recorded brain activity into images. Our end-to-end method for generating images from brain activity opens new possibilities for creative expression and experimentation. Notably, our results show that the generated images successfully reflect the dynamic and stochastic nature of the underlying neural activity, providing a unique procedure for visualization of brain function.

Visual tracking brain-computer interface

Brain-computer interfaces (BCIs) offer a way to interact with computers without relying on physical movements. Non-invasive electroencephalography-based visual BCIs, known for efficient speed and calibration ease, face limitations in continuous tasks due to discrete stimulus design and decoding methods. To achieve continuous control, we implemented a novel spatial encoding stimulus paradigm and devised a corresponding projection method to enable continuous modulation of decoded velocity. Subsequently, we conducted experiments involving 17 participants and achieved Fitt's information transfer rate (ITR) of 0.55 bps for the fixed tracking task and 0.37 bps for the random tracking task. The proposed BCI with a high Fitt's ITR was then integrated into two applications, including painting and gaming. In conclusion, this study proposed a visual BCI based-control method to go beyond discrete commands, allowing natural continuous control based on neural activity.

Brain-Computer interfaces for communication: preferences of individuals with locked-in syndrome, caregivers and researchers

OBJECTIVES

The development of Brain-Computer Interfaces to restore communication (cBCIs) in people with severe motor impairment ideally relies on a close collaboration between end-users and other stakeholders, such as caregivers and researchers. Awareness about potential differences in opinion between these groups is crucial for development of usable cBCIs and access technology (AT) in general. In this study, we compared the opinions of prospective cBCI users, their caregivers and cBCI researchers regarding: (1) what applications would users like to control with a cBCI; (2) what mental strategies would users prefer to use for cBCI control; and (3) at what stage of their clinical trajectory would users like to be informed about AT and cBCIs.

METHODS

We collected data from 28 individuals with locked-in syndrome, 29 of their caregivers and 28 cBCI researchers. The questionnaire was supported with animation videos to explain different cBCI concepts, the utility of which was also assessed.

RESULTS

Opinions of the three groups were aligned with respect to the most desired cBCI applications, but diverged regarding mental strategies and the timing of being informed about cBCIs. Animation videos were regarded as clear and useful tools to explain cBCIs and mental strategies to end-users and other stakeholders.

CONCLUSIONS

Disagreements were clear between stakeholders regarding which mental strategies users prefer to use and when they would like to be informed about cBCIs. To move forward in the development and clinical implementation of cBCIs, it will be necessary to align the research agendas with the needs of the end-users and caregivers.IMPLICATIONS FOR REHABILITATIONBrain-Computer Interfaces may offer people with severe motor impairment a brain-based and muscle-independent approach to control communication-technology. The successful development of communication BCIs (cBCIs) relies on a close collaboration between end-users and other stakeholders, such as caregivers and researchers.Our work reveals that people with locked-in syndrome (end-users), their caregivers and researchers developing cBCIs agree that direct and private forms of communication are the most desired cBCI applications, but disagree regarding the preferred mental strategies for cBCI control and when to be informed about cBCIs.Animation videos are an effective tool for providing information to individuals, independent of their level of health literacy, regarding the concept of cBCIs and mental strategies for control.The misalignment in opinions of different groups of stakeholders about cBCIs strengthens the argument for a user-centered design approach in the development of cBCIs and access technology designed for daily life usage.

A BCI painting system using a hybrid control approach based on SSVEP and P300

Brain-computer interfaces (BCIs) can help people with disabilities to communicate with others, express themselves, and even create art. In this paper, a BCI painting system using a hybrid control approach based on steady-state visual evoked potential (SSVEP) and P300 was developed, which can enable simple painting through brain-controlled painting tools. The BCI painting system is composed of two parts: a hybrid stimulus interface and a hybrid electroencephalogram (EEG) signal processing module. The user selects the menus and tools through the SSVEP and P300 stimulus matrices, respectively, and the paintings are displayed in the canvas area of the hybrid stimulus interface in real time. Twenty subjects participated in this study. An offline training experiment was performed to construct the P300 and SSVEP recognition models for each subject; an online painting experiment, which included a copy-painting task and a free-painting task, was performed to evaluate the BCI painting system. The results of the online painting experiment showed that the average tool selection accuracy (88.92 ± 3.94%) of the BCI painting system using the hybrid stimulus interface was slightly higher than that of the traditional brain painting system based on the P300 stimulus interface; the average information transfer rate (ITR) (74.20 ± 5.28 bpm, 71.80 ± 5.15 bpm) in the copy-painting and free-painting tasks of the BCI painting system was significantly higher than that of the traditional brain painting system. Our BCI painting system can effectively help users express their artistic creativity and improve their painting efficiency, and can provide new methods and new ideas for developing BCI-controlled applications.

Overview of the EEG-Based Classification of Motor Imagery Activities Using Machine Learning Methods and Inference Acceleration with FPGA-Based Cards

In this article, we provide a brief overview of the EEG-based classification of motor imagery activities using machine learning methods. We examined the effect of data segmentation and different neural network structures. By applying proper window size and using a purely convolutional neural network, we achieved 97.7% recognition accuracy on data from twenty subjects in three classes. The proposed architecture outperforms several networks used in previous research and makes the motor imagery-based BCI more efficient in some applications. In addition, we examined the performance of the neural network on a FPGA-based card and compared it with the inference speed and accuracy provided by a general-purpose processor.

Extending Brain-Computer Interface Access with a Multilingual Language Model in the P300 Speller

Brain-computer interfaces (BCI) such as the P300 speller have the potential to restore communication to advanced-stage neuromuscular disease patients. Research has improved typing speed and accuracy through innovations including the use of language models. While significant advances have been made, implementations have largely been restricted to a single language, primarily English. It is unclear whether these improvements would extend to other languages that present potential technical hurdles due to different alphabets and grammatical structures. Here, we adapt a language model-based classifier designed for English to two other languages, Spanish and Greek, to demonstrate the generalizability of these methods. Online experimental trials with 30 healthy native English, Spanish, and Greek speakers showed no significant difference between performances using the different versions of the system (66.20 vs. 61.97 vs. 60.89 bits/minute). Extending these methods across languages allows for expanding access to BCI systems to other populations, particularly in the developing world.

Brain-Computer Interfaces for Children With Complex Communication Needs and Limited Mobility: A Systematic Review

Brain-computer interfaces (BCIs) represent a new frontier in the effort to maximize the ability of individuals with profound motor impairments to interact and communicate. While much literature points to BCIs' promise as an alternative access pathway, there have historically been few applications involving children and young adults with severe physical disabilities. As research is emerging in this sphere, this article aims to evaluate the current state of translating BCIs to the pediatric population. A systematic review was conducted using the Scopus, PubMed, and Ovid Medline databases. Studies of children and adolescents that reported BCI performance published in English in peer-reviewed journals between 2008 and May 2020 were included. Twelve publications were identified, providing strong evidence for continued research in pediatric BCIs. Research evidence was generally at multiple case study or exploratory study level, with modest sample sizes. Seven studies focused on BCIs for communication and five on mobility. Articles were categorized and grouped based on type of measurement (i.e., non-invasive and invasive), and the type of brain signal (i.e., sensory evoked potentials or movement-related potentials). Strengths and limitations of studies were identified and used to provide requirements for clinical translation of pediatric BCIs. This systematic review presents the state-of-the-art of pediatric BCIs focused on developing advanced technology to support children and youth with communication disabilities or limited manual ability. Despite a few research studies addressing the application of BCIs for communication and mobility in children, results are encouraging and future works should focus on customizable pediatric access technologies based on brain activity.

Expert-Level Intracranial Electroencephalogram Ictal Pattern Detection by a Deep Learning Neural Network

Background: Decision-making in epilepsy surgery is strongly connected to the interpretation of the intracranial EEG (iEEG). Although deep learning approaches have demonstrated efficiency in processing extracranial EEG, few studies have addressed iEEG seizure detection, in part due to the small number of seizures per patient typically available from intracranial investigations. This study aims to evaluate the efficiency of deep learning methodology in detecting iEEG seizures using a large dataset of ictal patterns collected from epilepsy patients implanted with a responsive neurostimulation system (RNS). Methods: Five thousand two hundred and twenty-six ictal events were collected from 22 patients implanted with RNS. A convolutional neural network (CNN) architecture was created to provide personalized seizure annotations for each patient. Accuracy of seizure identification was tested in two scenarios: patients with seizures occurring following a period of chronic recording (scenario 1) and patients with seizures occurring immediately following implantation (scenario 2). The accuracy of the CNN in identifying RNS-recorded iEEG ictal patterns was evaluated against human neurophysiology expertise. Statistical performance was assessed via the area-under-precision-recall curve (AUPRC). Results: In scenario 1, the CNN achieved a maximum mean binary classification AUPRC of 0.84 ± 0.19 (95%CI, 0.72-0.93) and mean regression accuracy of 6.3 ± 1.0 s (95%CI, 4.3-8.5 s) at 30 seed samples. In scenario 2, maximum mean AUPRC was 0.80 ± 0.19 (95%CI, 0.68-0.91) and mean regression accuracy was 6.3 ± 0.9 s (95%CI, 4.8-8.3 s) at 20 seed samples. We obtained near-maximum accuracies at seed size of 10 in both scenarios. CNN classification failures can be explained by ictal electro-decrements, brief seizures, single-channel ictal patterns, highly concentrated interictal activity, changes in the sleep-wake cycle, and progressive modulation of electrographic ictal features. Conclusions: We developed a deep learning neural network that performs personalized detection of RNS-derived ictal patterns with expert-level accuracy. These results suggest the potential for automated techniques to significantly improve the management of closed-loop brain stimulation, including during the initial period of recording when the device is otherwise naïve to a given patient's seizures.

Brain-Computer Interfaces for Communication: Preferences of Individuals With Locked-in Syndrome

BACKGROUND

Brain-computer interfaces (BCIs) have been proposed as an assistive technology (AT) allowing people with locked-in syndrome (LIS) to use neural signals to communicate. To design a communication BCI (cBCI) that is fully accepted by the users, their opinion should be taken into consideration during the research and development process.

OBJECTIVE

We assessed the preferences of prospective cBCI users regarding (1) the applications they would like to control with a cBCI, (2) the mental strategies they would prefer to use to control the cBCI, and (3) when during their clinical trajectory they would like to be informed about AT and cBCIs. Furthermore, we investigated if individuals diagnosed with progressive and sudden onset (SO) disorders differ in their opinion.

METHODS

We interviewed 28 Dutch individuals with LIS during a 3-hour home visit using multiple-choice, ranking, and open questions. During the interview, participants were informed about BCIs and the possible mental strategies.

RESULTS

Participants rated (in)direct forms of communication, computer use, and environmental control as the most desired cBCI applications. In addition, active cBCI control strategies were preferred over reactive strategies. Furthermore, individuals with progressive and SO disorders preferred to be informed about AT and cBCIs at the moment they would need it.

CONCLUSIONS

We show that individuals diagnosed with progressive and SO disorders preferred, in general, the same applications, mental strategies, and time of information. By collecting the opinion of a large sample of individuals with LIS, this study provides valuable information to stakeholders in cBCI and other AT development.

Recommendations for Integrating a P300-Based Brain–Computer Interface in Virtual Reality Environments for Gaming: An Update

The integration of a P300-based brain–computer interface (BCI) into virtual reality (VR) environments is promising for the video games industry. However, it faces several limitations, mainly due to hardware constraints and limitations engendered by the stimulation needed by the BCI. The main restriction is still the low transfer rate that can be achieved by current BCI technology, preventing movement while using VR. The goal of this paper is to review current limitations and to provide application creators with design recommendations to overcome them, thus significantly reducing the development time and making the domain of BCI more accessible to developers. We review the design of video games from the perspective of BCI and VR with the objective of enhancing the user experience. An essential recommendation is to use the BCI only for non-complex and non-critical tasks in the game. Also, the BCI should be used to control actions that are naturally integrated into the virtual world. Finally, adventure and simulation games, especially if cooperative (multi-user), appear to be the best candidates for designing an effective VR game enriched by BCI technology.

A comprehensive assessment of Brain Computer Interfaces: Recent trends and challenges

BACKGROUND

An uninterrupted channel of communication and control between the human brain and electronic processing units has led to an increased use of Brain Computer Interfaces (BCIs). This article attempts to present an all-encompassing review on BCI and the scientific advancements associated with it. The ultimate goal of this review is to provide a general overview of the BCI technology and to shed light on different aspects of BCIs. This review also underscores the applications, practical challenges and opportunities associated with BCI technology, which can be used to accelerate future developments in this field.

METHODS

This review is based on a systematic literature search for tracking down the relevant research annals and proceedings. Using a methodical search strategy, the search was carried out across major technical databases. The retrieved records were screened for their relevance and a total of 369 research chronicles were engulfed in this review based on the inclusion criteria.

RESULTS

This review describes the present scenario and recent advancements in BCI technology. It also identifies several application areas of BCI technology. This comprehensive review provides evidence that, while we are getting ever closer, significant challenges still exist for the development of BCIs that can seamlessly integrate with the user's biological system.

CONCLUSION

The findings of this review confirm the importance of BCI technology in various applications. It is concluded that BCI technology, still in its sprouting phase, requires significant explorations for further development.

30+ years of P300 brain-computer interfaces

Brain-computer interfaces (BCIs) directly measure brain activity with no physical movement and translate the neural signals into messages. BCIs that employ the P300 event-related brain potential often have used the visual modality. The end user is presented with flashing stimuli that indicate selections for communication, control, or both. Counting each flash that corresponds to a specific target selection while ignoring other flashes will elicit P300s to only the target selection. P300 BCIs also have been implemented using auditory or tactile stimuli. P300 BCIs have been used with a variety of applications for severely disabled end users in their homes without frequent expert support. P300 BCI research and development has made substantial progress, but challenges remain before these tools can become practical devices for impaired patients and perhaps healthy people.

Comparison of the ERP-Based BCI Performance Among Chromatic (RGB) Semitransparent Face Patterns

Objective

Previous studies have shown that combing with color properties may be used as part of the display presented to BCI users in order to improve performance. Build on this, we explored the effects of combinations of face stimuli with three primary colors (RGB) on BCI performance which is assessed by classification accuracy and information transfer rate (ITR). Furthermore, we analyzed the waveforms of three patterns.

Methods

We compared three patterns in which semitransparent face is overlaid three primary colors as stimuli: red semitransparent face (RSF), green semitransparent face (GSF), and blue semitransparent face (BSF). Bayesian linear discriminant analysis (BLDA) was used to construct the individual classifier model. In addition, a Repeated-measures ANOVA (RM-ANOVA) and Bonferroni correction were chosen for statistical analysis.

Results

The results indicated that the RSF pattern achieved the highest online averaged accuracy with 93.89%, followed by the GSF pattern with 87.78%, while the lowest performance was caused by the BSF pattern with an accuracy of 81.39%. Furthermore, significant differences in classification accuracy and ITR were found between RSF and GSF (p < 0.05) and between RSF and BSF patterns (p < 0.05).

Conclusion

The semitransparent faces colored red (RSF) pattern yielded the best performance of the three patterns. The proposed patterns based on ERP-BCI system have a clinically significant impact by increasing communication speed and accuracy of the P300-speller for patients with severe motor impairment.

What is it like to use a BCI? - insights from an interview study with brain-computer interface users

Background

The neurotechnology behind brain-computer interfaces (BCIs) raises various ethical questions. The ethical literature has pinpointed several issues concerning safety, autonomy, responsibility and accountability, psychosocial identity, consent, privacy and data security. This study aims to assess BCI users’ experiences, self-observations and attitudes in their own right and looks for social and ethical implications.

Methods

We conducted nine semi-structured interviews with BCI users, who used the technology for medical reasons. The transcribed interviews were analyzed according to the Grounded Theory coding method.

Results

BCI users perceive themselves as active operators of a technology that offers them social participation and impacts their self-definition. Each of these aspects bears its own opportunities and risks. BCIs can contribute to retaining or regaining human capabilities. At the same time, BCI use contains elements that challenge common experiences, for example when the technology is in conflict with the affective side of BCI users. The potential benefits of BCIs are regarded as outweighing the risks in that BCI use is considered to promote valuable qualities and capabilities. BCI users appreciate the opportunity to regain lost capabilities as well as to gain new ones.

Conclusions

BCI users appreciate the technology for various reasons. The technology is highly appreciated in cases where it is beneficial in terms of agency, participation and self-definitions. Rather than questioning human nature, the technology can retain and restore characteristics and abilities which enrich our lives.

Workshops of the Seventh International Brain-Computer Interface Meeting: Not Getting Lost in Translation

The Seventh International Brain-Computer Interface (BCI) Meeting was held May 21-25th, 2018 at the Asilomar Conference Grounds, Pacific Grove, California, United States. The interactive nature of this conference was embodied by 25 workshops covering topics in BCI (also called brain-machine interface) research. Workshops covered foundational topics such as hardware development and signal analysis algorithms, new and imaginative topics such as BCI for virtual reality and multi-brain BCIs, and translational topics such as clinical applications and ethical assumptions of BCI development. BCI research is expanding in the diversity of applications and populations for whom those applications are being developed. BCI applications are moving toward clinical readiness as researchers struggle with the practical considerations to make sure that BCI translational efforts will be successful. This paper summarizes each workshop, providing an overview of the topic of discussion, references for additional information, and identifying future issues for research and development that resulted from the interactions and discussion at the workshop.

P300 Speller Performance Predictor Based on RSVP Multi-feature

Brain-computer interface (BCI) systems were developed so that people can control computers or machines through their brain activity without moving their limbs. The P300 speller is one of the BCI applications used most commonly, as is very simple and reliable and can achieve satisfactory performance. However, like other BCIs, the P300 speller still has room for improvements in terms of its practical use, for example, selecting the best compromise between spelling accuracy and information transfer rate (ITR; speed) so that the P300 speller can maintain high accuracy while increasing spelling speed. Therefore, seeking correlates of, and predicting, the P300 speller's performance is necessary to understand and improve the technique. In this work, we investigated the correlations between rapid serial visual presentation (RSVP) task features and the P300 speller's performance. Fifty-five subjects participated in the RSVP and conventional matrix P300 speller tasks and RSVP behavioral and electroencephalography (EEG) features were compared in the P300's speller performance. We found that several of the RSVP's event-related potential (ERP) and behavioral features were correlated with the P300 speller's offline binary classification accuracy. Using these features, we propose a simple multi-feature performance predictor (r = 0.53, p = 0.0001) that outperforms any single feature performance predictor, including that of the conventional RSVP T1% predictor (r = 0.28, p = 0.06). This result demonstrates that selective multi-features can predict BCI performance better than a single feature alone.

Cursive Eye-Writing With Smooth-Pursuit Eye-Movement Is Possible in Subjects With Amyotrophic Lateral Sclerosis

Amyotrophic lateral sclerosis (ALS) is a neurodegenerative disorder causing a progressive motor weakness of all voluntary muscles, whose progression challenges communication modalities such as handwriting or speech. The current study investigated whether ALS subjects can use Eye-On-Line (EOL), a novel eye-operated communication device allowing, after training, to voluntarily control smooth-pursuit eye-movements (SPEM) so as to eye-write in cursive. To that aim, ALS participants (n = 12) with preserved eye-movements but impaired handwriting were trained during six on-site visits. The primary outcome of the study was the recognition of eye-written digits (0-9) from ALS and healthy control subjects by naïve "readers." Changes in oculomotor performance and the safety of EOL were also evaluated. At the end of the program, 69.4% of the eye-written digits from 11 ALS subjects were recognized by naïve readers, similar to the 67.3% found for eye-written digits from controls participants, with however, large inter-individual differences in both groups of "writers." Training with EOL was associated with a transient fatigue leading one ALS subject to drop out the study at the fifth visit. Otherwise, itching eyes was the most common adverse event (3 subjects). This study shows that, despite the impact of ALS on the motor system, most ALS participants could improve their mastering of eye-movements, so as to produce recognizable eye-written digits, although the eye-traces sometimes needed smoothing to ease digit legibility from both ALS subjects and control participants. The capability to endogenously and voluntarily generate eye-traces using EOL brings a novel way to communicate for disabled individuals, allowing creative personal and emotional expression.

Wireless Computer-Supported Cooperative Work: A Pilot Experiment on Art and Brain⁻Computer Interfaces

The present case study looked into the feasibility of using brain–computer interface (BCI) technology combined with computer-supported cooperative work (CSCW) in a wireless network. We had two objectives; first, to test the wireless BCI-based configuration and the practical use of this idea we assessed workload perception in participants located several kilometers apart taking part in the same drawing task. Second, we studied the cortical activation patterns of participants performing the drawing task with and without the BCI technology. Results showed higher mental workload perception and broader cortical activation (frontal-temporal-occipital) under BCI experimental conditions. This idea shows a possible application of BCI research in the social field, where two or more users could engage in a computer networking task using BCI technology over the internet. New research avenues for CSCW are discussed and possibilities for future research are given.

Ethical Principles in Patient-Centered Medical Care to Support Quality of Life in Amyotrophic Lateral Sclerosis

It is one of the primary goals of medical care to secure good quality of life (QoL) while prolonging survival. This is a major challenge in severe medical conditions with a prognosis such as amyotrophic lateral sclerosis (ALS). Further, the definition of QoL and the question whether survival in this severe condition is compatible with a good QoL is a matter of subjective and culture-specific debate. Some people without neurodegenerative conditions believe that physical decline is incompatible with satisfactory QoL. Current data provide extensive evidence that psychosocial adaptation in ALS is possible, indicated by a satisfactory QoL. Thus, there is no fatalistic link of loss of QoL when physical health declines. There are intrinsic and extrinsic factors that have been shown to successfully facilitate and secure QoL in ALS which will be reviewed in the following article following the four ethical principles (1) Beneficence, (2) Non-maleficence, (3) Autonomy and (4) Justice, which are regarded as key elements of patient centered medical care according to Beauchamp and Childress. This is a JPND-funded work to summarize findings of the project NEEDSinALS (www.NEEDSinALS.com) which highlights subjective perspectives and preferences in medical decision making in ALS.

Acceptability Study of A3-K3 Robotic Architecture for a Neurorobotics Painting

In this paper, authors present a novel architecture for controlling an industrial robot via Brain Computer Interface. The robot used is a Series 2000 KR 210-2. The robotic arm was fitted with DI drawing devices that clamp, hold and manipulate various artistic media like brushes, pencils, pens. User selected a high-level task, for instance a shape or movement, using a human machine interface and the translation in robot movement was entirely demanded to the Robot Control Architecture defining a plan to accomplish user's task. The architecture was composed by a Human Machine Interface based on P300 Brain Computer Interface and a robotic architecture composed by a deliberative layer and a reactive layer to translate user's high-level command in a stream of movement for robots joints. To create a real-case scenario, the architecture was presented at Ars Electronica Festival, where the A3-K3 architecture has been used for painting. Visitors completed a survey to address 4 self-assessed different dimensions related to human-robot interaction: the technology knowledge, the personal attitude, the innovativeness and the satisfaction. The obtained results have led to further exploring the border of human-robot interaction, highlighting the possibilities of human expression in the interaction process with a machine to create art.

Faces, locations, and tools: a proposed two-stimulus P300 brain computer interface

OBJECTIVE

Brain computer interface (BCI) technology can be important for those unable to communicate due to loss of muscle control. Given that the P300 Speller provides a relatively slow rate of communication, highly accurate classification is of great importance. Previous studies have shown that alternative stimuli (e.g. faces) can improve BCI speed and accuracy. The present study uses two new alternative stimuli, locations and graspable tools. Functional MRI studies have shown that images of familiar locations produce brain responses in the parahippocampal place area and graspable tools produce brain responses in premotor cortex.

APPROACH

The current studies show that location and tool stimuli produce unique and discriminable brain responses that can be used to improve offline classification accuracy. Experiment 1 presented face stimuli and location stimuli and Experiment 2 presented location and tool stimuli.

MAIN RESULTS

In both experiments, offline results showed that a stimulus specific classifier provided higher accuracy, speed, and bit rate.

SIGNIFICANCE

This study was used to provide preliminary offline support for using unique stimuli to improve speed and accuracy of the P300 Speller. Additional experiments should be conducted to examine the online efficacy of this novel paradigm.

Eyes-Closed Increases the Usability of Brain-Computer Interfaces Based on Auditory Event-Related Potentials

Recent research has demonstrated how brain-computer interfaces (BCI) based on auditory stimuli can be used for communication and rehabilitation. In these applications, users are commonly instructed to avoid eye movements while keeping their eyes open. This secondary task can lead to exhaustion and subjects may not succeed in suppressing eye movements. In this work, we investigate the option to use a BCI with eyes-closed. Twelve healthy subjects participated in a single electroencephalography (EEG) session where they were listening to a rapid stream of bisyllabic words while alternatively having their eyes open or closed. In addition, we assessed usability aspects for the two conditions with a questionnaire. Our analysis shows that eyes-closed does not reduce the number of eye artifacts and that event-related potential (ERP) responses and classification accuracies are comparable between both conditions. Importantly, we found that subjects expressed a significant general preference toward the eyes-closed condition and were also less tensed in that condition. Furthermore, switching between eyes-closed and eyes-open and vice versa is possible without a severe drop in classification accuracy. These findings suggest that eyes-closed should be considered as a viable alternative in auditory BCIs that might be especially useful for subjects with limited control over their eye movements.

Screening for Cognitive Function in Complete Immobility Using Brain-Machine Interfaces: A Proof of Principle Study

Background: In many neurological conditions, there is a combination of decline in physical function and cognitive abilities. For far advanced stages of physical disability where speaking and hand motor abilities are severely impaired, there is a lack of standardized approach to screen for cognitive profile. Methods:N = 40 healthy subjects were included in the study. For proof of principle, N = 6 ALS patients were additionally measured. For cognitive screening, we used the Edinburgh cognitive and behavioral ALS screen (ECAS) in the standard paper-and-pencil version. Additionally, we adapted the ECAS to a brain-machine interface (BMI) control module to screen for cognition in severely advanced patients. Results: There was a high congruency between BMI version and the paper-and-pencil version of the ECAS. Sensitivity and specificity of the ECAS-BMI were mostly high whereas stress and weariness for the patient were low. Discussion/Conclusion: We hereby present evidence that adaptation of a standardized neuropsychological test for BMI control is feasible. BMI driven neuropsychological test provides congruent results compared to standardized tests with a good specificity and sensitivity but low patient load.

A Brain-Computer Interface (BCI) system to use arbitrary Windows applications by directly controlling mouse and keyboard

A Brain-Computer Interface (BCI) allows to control a computer by brain activity only, without the need for muscle control. In this paper, we present an EEG-based BCI system based on code-modulated visual evoked potentials (c-VEPs) that enables the user to work with arbitrary Windows applications. Other BCI systems, like the P300 speller or BCI-based browsers, allow control of one dedicated application designed for use with a BCI. In contrast, the system presented in this paper does not consist of one dedicated application, but enables the user to control mouse cursor and keyboard input on the level of the operating system, thereby making it possible to use arbitrary applications. As the c-VEP BCI method was shown to enable very fast communication speeds (writing more than 20 error-free characters per minute), the presented system is the next step in replacing the traditional mouse and keyboard and enabling complete brain-based control of a computer.

EEG-Based Brain-Computer Interfaces for Communication and Rehabilitation of People with Motor Impairment: A Novel Approach of the 21 st Century

People with severe neurological impairments face many challenges in sensorimotor functions and communication with the environment; therefore they have increased demand for advanced, adaptive and personalized rehabilitation. During the last several decades, numerous studies have developed brain-computer interfaces (BCIs) with the goals ranging from providing means of communication to functional rehabilitation. Here we review the research on non-invasive, electroencephalography (EEG)-based BCI systems for communication and rehabilitation. We focus on the approaches intended to help severely paralyzed and locked-in patients regain communication using three different BCI modalities: slow cortical potentials, sensorimotor rhythms and P300 potentials, as operational mechanisms. We also review BCI systems for restoration of motor function in patients with spinal cord injury and chronic stroke. We discuss the advantages and limitations of these approaches and the challenges that need to be addressed in the future.

Deep learning with convolutional neural networks for EEG decoding and visualization

Deep learning with convolutional neural networks (deep ConvNets) has revolutionized computer vision through end-to-end learning, that is, learning from the raw data. There is increasing interest in using deep ConvNets for end-to-end EEG analysis, but a better understanding of how to design and train ConvNets for end-to-end EEG decoding and how to visualize the informative EEG features the ConvNets learn is still needed. Here, we studied deep ConvNets with a range of different architectures, designed for decoding imagined or executed tasks from raw EEG. Our results show that recent advances from the machine learning field, including batch normalization and exponential linear units, together with a cropped training strategy, boosted the deep ConvNets decoding performance, reaching at least as good performance as the widely used filter bank common spatial patterns (FBCSP) algorithm (mean decoding accuracies 82.1% FBCSP, 84.0% deep ConvNets). While FBCSP is designed to use spectral power modulations, the features used by ConvNets are not fixed a priori. Our novel methods for visualizing the learned features demonstrated that ConvNets indeed learned to use spectral power modulations in the alpha, beta, and high gamma frequencies, and proved useful for spatially mapping the learned features by revealing the topography of the causal contributions of features in different frequency bands to the decoding decision. Our study thus shows how to design and train ConvNets to decode task-related information from the raw EEG without handcrafted features and highlights the potential of deep ConvNets combined with advanced visualization techniques for EEG-based brain mapping. Hum Brain Mapp 38:5391-5420, 2017. © 2017 Wiley Periodicals, Inc.

Composing only by thought: Novel application of the P300 brain-computer interface

The P300 event-related potential is a well-known pattern in the electroencephalogram (EEG). This kind of brain signal is used for many different brain-computer interface (BCI) applications, e.g., spellers, environmental controllers, web browsers, or for painting. In recent times, BCI systems are mature enough to leave the laboratories to be used by the end-users, namely severely disabled people. Therefore, new challenges arise and the systems should be implemented and evaluated according to user-centered design (USD) guidelines. We developed and implemented a new system that utilizes the P300 pattern to compose music. Our Brain Composing system consists of three parts: the EEG acquisition device, the P300-based BCI, and the music composing software. Seventeen musical participants and one professional composer performed a copy-spelling, a copy-composing, and a free-composing task with the system. According to the USD guidelines, we investigated the efficiency, the effectiveness and subjective criteria in terms of satisfaction, enjoyment, frustration, and attractiveness. The musical participants group achieved high average accuracies: 88.24% (copy-spelling), 88.58% (copy-composing), and 76.51% (free-composing). The professional composer achieved also high accuracies: 100% (copy-spelling), 93.62% (copy-composing), and 98.20% (free-composing). General results regarding the subjective criteria evaluation were that the participants enjoyed the usage of the Brain Composing system and were highly satisfied with the system. Showing very positive results with healthy people in this study, this was the first step towards a music composing system for severely disabled people.

The Role of the Interplay between Stimulus Type and Timing in Explaining BCI-Illiteracy for Visual P300-Based Brain-Computer Interfaces

Visual P300-based Brain-Computer Interface (BCI) spellers enable communication or interaction with the environment by flashing elements in a matrix and exploiting consequent changes in end-user's brain activity. Despite research efforts, performance variability and BCI-illiteracy still are critical issues for real world applications. Moreover, there is a quite unaddressed kind of BCI-illiteracy, which becomes apparent when the same end-user operates BCI-spellers intended for different applications: our aim is to understand why some well performers can become BCI-illiterate depending on speller type. We manipulated stimulus type (factor STIM: either characters or icons), color (factor COLOR: white, green) and timing (factor SPEED: fast, slow). Each BCI session consisted of training (without feedback) and performance phase (with feedback), both in copy-spelling. For fast flashing spellers, we observed a performance worsening for white icon-speller. Our findings are consistent with existing results reported on end-users using identical white×fast spellers, indicating independence of worsening trend from users' group. The use of slow stimulation timing shed a new light on the perceptual and cognitive phenomena related to the use of a BCI-speller during both the training and the performance phase. We found a significant STIM main effect for the N1 component on P z and PO7 during the training phase and on PO8 during the performance phase, whereas in both phases neither the STIM×COLOR interaction nor the COLOR main effect was statistically significant. After collapsing data for factor COLOR, it emerged a statistically significant modulation of N1 amplitude depending to the phase of BCI session: N1 was more negative for icons than for characters both on P z and PO7 (training), whereas the opposite modulation was observed for PO8 (performance). Results indicate that both feedback and expertise with respect to the stimulus type can modulate the N1 component and that icons require more perceptual analysis. Therefore, fast flashing is likely to be more detrimental for end-users' performance in case of icon-spellers. In conclusion, the interplay between stimulus type and timing seems relevant for a satisfactory and efficient end-user's BCI-experience.

Chronic disorders of consciousness

Over the last 20 years, studies have provided greater insight into disorders of consciousness (DOC), also known as altered state of consciousness. Increased brain residual functions have been identified in patients with DOC due to the successful application of novel next-generation imaging technologies. Many unconscious patients have now been confirmed to retain considerable cognitive functions. It is hoped that greater insight regarding the psychological state of patients may be achieved through the use of functional magnetic resonance imaging and brain-computer interfaces. However, issues surrounding the research and treatment of DOC remain problematic. These include differing opinions on the definition of consciousness, difficulties in diagnosis, assessment, prognosis and/or treatment, and newly emerging ethical, legal and social issues. To overcome these, appropriate care must be offered to patients with DOC by clinicians and families, as DOC patients may now be considered to live in more than just a vegetative state. The present article reviews the controversy surrounding the definition of consciousness and the reliability of novel technologies, prognostic prediction, communication with DOC patients and treatment methods. The ethical and social issues surrounding the treatment of DOC and future perspectives are also considered.

A Multifunctional Brain-Computer Interface Intended for Home Use: An Evaluation with Healthy Participants and Potential End Users with Dry and Gel-Based Electrodes

Current brain-computer interface (BCIs) software is often tailored to the needs of scientists and technicians and therefore complex to allow for versatile use. To facilitate home use of BCIs a multifunctional P300 BCI with a graphical user interface intended for non-expert set-up and control was designed and implemented. The system includes applications for spelling, web access, entertainment, artistic expression and environmental control. In addition to new software, it also includes new hardware for the recording of electroencephalogram (EEG) signals. The EEG system consists of a small and wireless amplifier attached to a cap that can be equipped with gel-based or dry contact electrodes. The system was systematically evaluated with a healthy sample, and targeted end users of BCI technology, i.e., people with a varying degree of motor impairment tested the BCI in a series of individual case studies. Usability was assessed in terms of effectiveness, efficiency and satisfaction. Feedback of users was gathered with structured questionnaires. Two groups of healthy participants completed an experimental protocol with the gel-based and the dry contact electrodes (N = 10 each). The results demonstrated that all healthy participants gained control over the system and achieved satisfactory to high accuracies with both gel-based and dry electrodes (average error rates of 6 and 13%). Average satisfaction ratings were high, but certain aspects of the system such as the wearing comfort of the dry electrodes and design of the cap, and speed (in both groups) were criticized by some participants. Six potential end users tested the system during supervised sessions. The achieved accuracies varied greatly from no control to high control with accuracies comparable to that of healthy volunteers. Satisfaction ratings of the two end-users that gained control of the system were lower as compared to healthy participants. The advantages and disadvantages of the BCI and its applications are discussed and suggestions are presented for improvements to pave the way for user friendly BCIs intended to be used as assistive technology by persons with severe paralysis.

Learning from label proportions in brain-computer interfaces: Online unsupervised learning with guarantees

Objective

Using traditional approaches, a brain-computer interface (BCI) requires the collection of calibration data for new subjects prior to online use. Calibration time can be reduced or eliminated e.g., by subject-to-subject transfer of a pre-trained classifier or unsupervised adaptive classification methods which learn from scratch and adapt over time. While such heuristics work well in practice, none of them can provide theoretical guarantees. Our objective is to modify an event-related potential (ERP) paradigm to work in unison with the machine learning decoder, and thus to achieve a reliable unsupervised calibrationless decoding with a guarantee to recover the true class means.

Method

We introduce learning from label proportions (LLP) to the BCI community as a new unsupervised, and easy-to-implement classification approach for ERP-based BCIs. The LLP estimates the mean target and non-target responses based on known proportions of these two classes in different groups of the data. We present a visual ERP speller to meet the requirements of LLP. For evaluation, we ran simulations on artificially created data sets and conducted an online BCI study with 13 subjects performing a copy-spelling task.

Results

Theoretical considerations show that LLP is guaranteed to minimize the loss function similar to a corresponding supervised classifier. LLP performed well in simulations and in the online application, where 84.5% of characters were spelled correctly on average without prior calibration.

Significance

The continuously adapting LLP classifier is the first unsupervised decoder for ERP BCIs guaranteed to find the optimal decoder. This makes it an ideal solution to avoid tedious calibration sessions. Additionally, LLP works on complementary principles compared to existing unsupervised methods, opening the door for their further enhancement when combined with LLP.

P300-Based Asynchronous Brain Computer Interface for Environmental Control System

An asynchronous brain computer interface (A-BCI) determines whether or not a subject is on control state, and produces control commands only in case of subject's being on control state. In this study, we propose a novel P300-based A-BCI algorithm that distinguishes control state and noncontrol state of users. Furthermore, A-BCI algorithm combined with a dynamic stopping function that enables users to select control command independent from a fixed number of intensification sequence. The proposed P300-based A-BCI algorithm uses classification patterns to determine control state and uses optimal operating point of receiver operating characteristics curve for dynamic stopping function. The proposed A-BCI algorithm is also combined with region-based paradigm (RBP) based stimulus interface. The A-BCI algorithm is tested on an internet-based environmental control system. A total of ten nondisabled subjects were participated voluntarily in the experiments. Two-level approach of the RBP-based stimulus interface improves noncontrol state detection accuracy up to 100%. Besides, ratio of incorrect command selection at control state is reduced significantly. At control state, ratio of correct selections, incorrect selections, and missed selections are 93.27%, 1.09%, and 5.63%, respectively. On the other hand, dynamic stopping function enables command selections at least two intensifications. Mean number of intensification sequences to complete the given tasks is 3.15. Thanks to dynamic stopping function, it provides a significant improvement on information transfer rate. The proposed A-BCI algorithm is important in terms of practical BCI systems.

Reaching and Grasping a Glass of Water by Locked-In ALS Patients through a BCI-Controlled Humanoid Robot

Locked-in Amyotrophic Lateral Sclerosis (ALS) patients are fully dependent on caregivers for any daily need. At this stage, basic communication and environmental control may not be possible even with commonly used augmentative and alternative communication devices. Brain Computer Interface (BCI) technology allows users to modulate brain activity for communication and control of machines and devices, without requiring a motor control. In the last several years, numerous articles have described how persons with ALS could effectively use BCIs for different goals, usually spelling. In the present study, locked-in ALS patients used a BCI system to directly control the humanoid robot NAO (Aldebaran Robotics, France) with the aim of reaching and grasping a glass of water. Four ALS patients and four healthy controls were recruited and trained to operate this humanoid robot through a P300-based BCI. A few minutes training was sufficient to efficiently operate the system in different environments. Three out of the four ALS patients and all controls successfully performed the task with a high level of accuracy. These results suggest that BCI-operated robots can be used by locked-in ALS patients as an artificial alter-ego, the machine being able to move, speak and act in his/her place.

High performance communication by people with paralysis using an intracortical brain-computer interface

Brain-computer interfaces (BCIs) have the potential to restore communication for people with tetraplegia and anarthria by translating neural activity into control signals for assistive communication devices. While previous pre-clinical and clinical studies have demonstrated promising proofs-of-concept (Serruya et al., 2002; Simeral et al., 2011; Bacher et al., 2015; Nuyujukian et al., 2015; Aflalo et al., 2015; Gilja et al., 2015; Jarosiewicz et al., 2015; Wolpaw et al., 1998; Hwang et al., 2012; Spüler et al., 2012; Leuthardt et al., 2004; Taylor et al., 2002; Schalk et al., 2008; Moran, 2010; Brunner et al., 2011; Wang et al., 2013; Townsend and Platsko, 2016; Vansteensel et al., 2016; Nuyujukian et al., 2016; Carmena et al., 2003; Musallam et al., 2004; Santhanam et al., 2006; Hochberg et al., 2006; Ganguly et al., 2011; O'Doherty et al., 2011; Gilja et al., 2012), the performance of human clinical BCI systems is not yet high enough to support widespread adoption by people with physical limitations of speech. Here we report a high-performance intracortical BCI (iBCI) for communication, which was tested by three clinical trial participants with paralysis. The system leveraged advances in decoder design developed in prior pre-clinical and clinical studies (Gilja et al., 2015; Kao et al., 2016; Gilja et al., 2012). For all three participants, performance exceeded previous iBCIs (Bacher et al., 2015; Jarosiewicz et al., 2015) as measured by typing rate (by a factor of 1.4-4.2) and information throughput (by a factor of 2.2-4.0). This high level of performance demonstrates the potential utility of iBCIs as powerful assistive communication devices for people with limited motor function.Clinical Trial No: NCT00912041.

Brain-computer interfaces for patients with disorders of consciousness

The disorders of consciousness refer to clinical conditions that follow a severe head injury. Patients diagnosed as in a vegetative state lack awareness, while patients diagnosed as in a minimally conscious state retain fluctuating awareness. However, it is a challenge to accurately diagnose these disorders with clinical assessments of behavior. To improve diagnostic accuracy, neuroimaging-based approaches have been developed to detect the presence or absence of awareness in patients who lack overt responsiveness. For the small subset of patients who retain awareness, brain-computer interfaces could serve as tools for communication and environmental control. Here we review the existing literature concerning the sensory and cognitive abilities of patients with disorders of consciousness with respect to existing brain-computer interface designs. We highlight the challenges of device development for this special population and address some of the most promising approaches for future investigations.

Brain Computer Interface on Track to Home

The novel BackHome system offers individuals with disabilities a range of useful services available via brain-computer interfaces (BCIs), to help restore their independence. This is the time such technology is ready to be deployed in the real world, that is, at the target end users' home. This has been achieved by the development of practical electrodes, easy to use software, and delivering telemonitoring and home support capabilities which have been conceived, implemented, and tested within a user-centred design approach. The final BackHome system is the result of a 3-year long process involving extensive user engagement to maximize effectiveness, reliability, robustness, and ease of use of a home based BCI system. The system is comprised of ergonomic and hassle-free BCI equipment; one-click software services for Smart Home control, cognitive stimulation, and web browsing; and remote telemonitoring and home support tools to enable independent home use for nonexpert caregivers and users. BackHome aims to successfully bring BCIs to the home of people with limited mobility to restore their independence and ultimately improve their quality of life.