Self-initiation of EEG-based brain-computer communication using the heart rate response

Effects of Jaw Clench Actions on Steady-State Visual Evoked Potential Detection at Some Typical Frequencies

More and more hybrid brain-computer interfaces (BCI) supplement traditional single-modality BCI in practical applications. Combinations based on steady-state visual evoked potential (SSVEP) and electromyography (EMG) are the widely used hybrid BCIs. The EMG of jaw clench is commonly used together with SSVEP. This article explored the interference with SSVEP from occipital electrodes by the jaw clench-related EMG so that SSVEP with specific frequency can be identified even during occlusal movements. The experiment was divided into three sets base on the jaw clench patterns (no clenches, chew, and long clench). In each set, the subjects used the same visual stimuli, which were realized by the three flashing targets at different frequencies (6.2Hz, 9.8Hz, and 14.6Hz). After collecting the SSVEP at 4 sites in the occipital region, the SSVEP response spectrum of each stimulus was observed under the three jaw clench patterns. Then, the SSVEP signal was identified by the canonical correlation analysis method for accuracy statistics. Spectrum responses showed that the interference of the jaw clench EMG on SSVEP could be avoided when the stimulation frequency is lower than 20Hz. SSVEP could be identified based on the frequency domain characteristics of these signals. During steady-state visual stimulation with jaw clenches, the recognition rate of SSVEP was still high (no clenches: 100.0%, chew: 94.7%, and long clench: 100.0%). Through reasonable frequency selecting and signal processing, the influence of the jaw clench movement on the SSVEP could be reduced and a high recognition accuracy could be achieved, even the jaw clench actions and the SSVEP stimulation occur simultaneously.

Brain Computer Interfaces for Improving the Quality of Life of Older Adults and Elderly Patients

All people experience aging, and the related physical and health changes, including changes in memory and brain function. These changes may become debilitating leading to an increase in dependence as people get older. Many external aids and tools have been developed to allow older adults and elderly patients to continue to live normal and comfortable lives. This mini-review describes some of the recent studies on cognitive decline and motor control impairment with the goal of advancing non-invasive brain computer interface (BCI) technologies to improve health and wellness of older adults and elderly patients. First, we describe the state of the art in cognitive prosthetics for psychiatric diseases. Then, we describe the state of the art of possible assistive BCI applications for controlling an exoskeleton, a wheelchair and smart home for elderly people with motor control impairments. The basic age-related brain and body changes, the effects of age on cognitive and motor abilities, and several BCI paradigms with typical tasks and outcomes are thoroughly described. We also discuss likely future trends and technologies to assist healthy older adults and elderly patients using innovative BCI applications with minimal technical oversight.

30+ years of P300 brain-computer interfaces

Brain-computer interfaces (BCIs) directly measure brain activity with no physical movement and translate the neural signals into messages. BCIs that employ the P300 event-related brain potential often have used the visual modality. The end user is presented with flashing stimuli that indicate selections for communication, control, or both. Counting each flash that corresponds to a specific target selection while ignoring other flashes will elicit P300s to only the target selection. P300 BCIs also have been implemented using auditory or tactile stimuli. P300 BCIs have been used with a variety of applications for severely disabled end users in their homes without frequent expert support. P300 BCI research and development has made substantial progress, but challenges remain before these tools can become practical devices for impaired patients and perhaps healthy people.

Physiological Responses During Hybrid BNCI Control of an Upper-Limb Exoskeleton

When combined with assistive robotic devices, such as wearable robotics, brain/neural-computer interfaces (BNCI) have the potential to restore the capabilities of handicapped people to carry out activities of daily living. To improve applicability of such systems, workload and stress should be reduced to a minimal level. Here, we investigated the user's physiological reactions during the exhaustive use of the interfaces of a hybrid control interface. Eleven BNCI-naive healthy volunteers participated in the experiments. All participants sat in a comfortable chair in front of a desk and wore a whole-arm exoskeleton as well as wearable devices for monitoring physiological, electroencephalographic (EEG) and electrooculographic (EoG) signals. The experimental protocol consisted of three phases: (i) Set-up, calibration and BNCI training; (ii) Familiarization phase; and (iii) Experimental phase during which each subject had to perform EEG and EoG tasks. After completing each task, the NASA-TLX questionnaire and self-assessment manikin (SAM) were completed by the user. We found significant differences (p-value < 0.05) in heart rate variability (HRV) and skin conductance level (SCL) between participants during the use of the two different biosignal modalities (EEG, EoG) of the BNCI. This indicates that EEG control is associated with a higher level of stress (associated with a decrease in HRV) and mental work load (associated with a higher level of SCL) when compared to EoG control. In addition, HRV and SCL modulations correlated with the subject's workload perception and emotional responses assessed through NASA-TLX questionnaires and SAM.

Exploring Cognitive Flexibility With a Noninvasive BCI Using Simultaneous Steady-State Visual Evoked Potentials and Sensorimotor Rhythms

EEG-based brain-computer interface (BCI) technology creates non-biological pathways for conveying a user's mental intent solely through noninvasively measured neural signals. While optimizing the performance of a single task has long been the focus of BCI research, in order to translate this technology into everyday life, realistic situations, in which multiple tasks are performed simultaneously, must be investigated. In this paper, we explore the concept of cognitive flexibility, or multitasking, within the BCI framework by utilizing a 2-D cursor control task, using sensorimotor rhythms (SMRs), and a four-target visual attention task, using steady-state visual evoked potentials (SSVEPs), both individually and simultaneously. We found no significant difference between the accuracy of the tasks when executing them alone (SMR-57.9% ± 15.4% and SSVEP-59.0% ± 14.2%) and simultaneously (SMR-54.9% ± 17.2% and SSVEP-57.5% ± 15.4%). These modest decreases in performance were supported by similar, non-significant changes in the electrophysiology of the SSVEP and SMR signals. In this sense, we report that multiple BCI tasks can be performed simultaneously without a significant deterioration in performance; this finding will help drive these systems toward realistic daily use in which a user's cognition will need to be involved in multiple tasks at once.

Combining Mental Training and Physical Training With Goal-Oriented Protocols in Stroke Rehabilitation: A Feasibility Case Study

Stroke is one of the leading causes of permanent disability in adults. The literature suggests that rehabilitation is key to early motor recovery. However, conventional therapy is labor and cost intensive. Robotic and functional electrical stimulation (FES) devices can provide a high dose of repetitions and as such may provide an alternative, or an adjunct, to conventional rehabilitation therapy. Brain-computer interfaces (BCI) could augment neuroplasticity by introducing mental training. However, mental training alone is not enough; but combining mental with physical training could boost outcomes. In the current case study, a portable rehabilitative platform and goal-oriented supporting training protocols were introduced and tested with a chronic stroke participant. A novel training method was introduced with the proposed rehabilitative platform. A 37-year old individual with chronic stroke participated in 6-weeks of training (18 sessions in total, 3 sessions a week, and 1 h per session). In this case study, we show that an individual with chronic stroke can tolerate a 6-week training bout with our system and protocol. The participant was actively engaged throughout the training. Changes in the Wolf Motor Function Test (WMFT) suggest that the training positively affected arm motor function (12% improvement in WMFT score).

A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives

A new Brain-Computer Interface (BCI) technique, which is called a hybrid BCI, has recently been proposed to address the limitations of conventional single BCI system. Although some hybrid BCI studies have shown promising results, the field of hybrid BCI is still in its infancy and there is much to be done. Especially, since the hybrid BCI systems are so complicated and complex, it is difficult to understand the constituent and role of a hybrid BCI system at a glance. Also, the complicated and complex systems make it difficult to evaluate the usability of the systems. We systematically reviewed and analyzed the current state-of-the-art hybrid BCI studies, and proposed a systematic taxonomy for classifying the types of hybrid BCIs with multiple taxonomic criteria. After reviewing 74 journal articles, hybrid BCIs could be categorized with respect to 1) the source of brain signals, 2) the characteristics of the brain signal, and 3) the characteristics of operation in each system. In addition, we exhaustively reviewed recent literature on usability of BCIs. To identify the key evaluation dimensions of usability, we focused on task and measurement characteristics of BCI usability. We classified and summarized 31 BCI usability journal articles according to task characteristics (type and description of task) and measurement characteristics (subjective and objective measures). Afterwards, we proposed usability dimensions for BCI and hybrid BCI systems according to three core-constructs: Satisfaction, effectiveness, and efficiency with recommendations for further research. This paper can help BCI researchers, even those who are new to the field, can easily understand the complex structure of the hybrid systems at a glance. Recommendations for future research can also be helpful in establishing research directions and gaining insight in how to solve ergonomics and HCI design issues surrounding BCI and hybrid BCI systems by usability evaluation.

User Experience May be Producing Greater Heart Rate Variability than Motor Imagery Related Control Tasks during the User-System Adaptation in Brain-Computer Interfaces

Brain-computer interface (BCI) is technology that is developing fast, but it remains inaccurate, unreliable and slow due to the difficulty to obtain precise information from the brain. Consequently, the involvement of other biosignals to decode the user control tasks has risen in importance. A traditional way to operate a BCI system is via motor imagery (MI) tasks. As imaginary movements activate similar cortical structures and vegetative mechanisms as a voluntary movement does, heart rate variability (HRV) has been proposed as a parameter to improve the detection of MI related control tasks. However, HR is very susceptible to body needs and environmental demands, and as BCI systems require high levels of attention, perceptual processing and mental workload, it is important to assess the practical effectiveness of HRV. The present study aimed to determine if brain and heart electrical signals (HRV) are modulated by MI activity used to control a BCI system, or if HRV is modulated by the user perceptions and responses that result from the operation of a BCI system (i.e., user experience). For this purpose, a database of 11 participants who were exposed to eight different situations was used. The sensory-cognitive load (intake and rejection tasks) was controlled in those situations. Two electrophysiological signals were utilized: electroencephalography and electrocardiography. From those biosignals, event-related (de-)synchronization maps and event-related HR changes were respectively estimated. The maps and the HR changes were cross-correlated in order to verify if both biosignals were modulated due to MI activity. The results suggest that HR varies according to the experience undergone by the user in a BCI working environment, and not because of the MI activity used to operate the system.

Noninvasive Brain-Computer Interfaces Based on Sensorimotor Rhythms

Brain-computer interfaces (BCIs) have been explored in the field of neuroengineering to investigate how the brain can use these systems to control external devices. We review the principles and approaches we have taken to develop a sensorimotor rhythm EEG based brain-computer interface (BCI). The methods include developing BCI systems incorporating the control of physical devices to increase user engagement, improving BCI systems by inversely mapping scalp-recorded EEG signals to the cortical source domain, integrating BCI with noninvasive neuromodulation strategies to improve learning, and incorporating mind-body awareness training to enhance BCI learning and performance. The challenges and merits of these strategies are discussed, together with recent findings. Our work indicates that the sensorimotor-rhythm-based noninvasive BCI has the potential to provide communication and control capabilities as an alternative to physiological motor pathways.

Achieving a hybrid brain-computer interface with tactile selective attention and motor imagery

OBJECTIVE

We propose a new hybrid brain-computer interface (BCI) system that integrates two different EEG tasks: tactile selective attention (TSA) using a vibro-tactile stimulator on the left/right finger and motor imagery (MI) of left/right hand movement. Event-related desynchronization (ERD) from the MI task and steady-state somatosensory evoked potential (SSSEP) from the TSA task are retrieved and combined into two hybrid senses.

APPROACH

One hybrid approach is to measure two tasks simultaneously; the features of each task are combined for testing. Another hybrid approach is to measure two tasks consecutively (TSA first and MI next) using only MI features. For comparison with the hybrid approaches, the TSA and MI tasks are measured independently.

MAIN RESULTS

Using a total of 16 subject datasets, we analyzed the BCI classification performance for MI, TSA and two hybrid approaches in a comparative manner; we found that the consecutive hybrid approach outperformed the others, yielding about a 10% improvement in classification accuracy relative to MI alone. It is understood that TSA may play a crucial role as a prestimulus in that it helps to generate earlier ERD prior to MI and thus sustains ERD longer and to a stronger degree; this ERD may give more discriminative information than ERD in MI alone.

SIGNIFICANCE

Overall, our proposed consecutive hybrid approach is very promising for the development of advanced BCI systems.

An analysis of performance evaluation for motor-imagery based BCI

In recent years, numerous brain-computer interfaces (BCIs) based on motor-imagery have been proposed which incorporate features such as adaptive classification, error detection and correction, fusion with auxiliary signals and shared control capabilities. Due to the added complexity of such algorithms, the evaluation strategy and metrics used for analysis must be carefully chosen to accurately represent the performance of the BCI. In this article, metrics are reviewed and contrasted using both simulated examples and experimental data. Furthermore, a review of the recent literature is presented to determine how BCIs are evaluated, in particular, focusing on the relationship between how the data are used relative to the BCI subcomponent under investigation. From the analysis performed in this study, valuable guidelines are presented regarding the choice of metrics and evaluation strategy dependent upon any chosen BCI paradigm.

Towards a BCI for sensorimotor training: initial results from simultaneous fNIRS and biosignal recordings

This paper presents the concept and initial results of a novel approach for robot assisted sensorimotor training in stroke rehabilitation. It is based on a brain-body-robot interface (B(2)RI), combining both neural and physiological recordings, that detects the intention to perform a motor task. By directly including the injured brain into the therapy, we ultimately aim at providing a new method for severely impaired patients to engage in active movement therapy. In the present study, seven healthy subjects performed an isometric finger pinching task while functional near-infrared spectroscopy (fNIRS) signals from motor cortical areas and biosignals were recorded simultaneously. Results showed an insignificant increase in the blood pressure during the preparation period prior to motor execution. During the execution period, significant changes in oxy-and deoxyhemoglobin were found in the primary motor cortex, accompanied by an increase in blood pressure, respiration rate and galvanic skin response (GSR). Cortical measurements of premotor areas and heart rate revealed significant changes at the subject level with large inter-subject variability. The results presented here will serve as priors for the design of further studies to test the efficacy of the concept with stroke patients, and the found effects will provide a basis for the development of a classifier for a future B(2)RI.

A hybrid brain-computer interface based on the fusion of electroencephalographic and electromyographic activities

Hybrid brain-computer interfaces (BCIs) are representing a recent approach to develop practical BCIs. In such a system disabled users are able to use all their remaining functionalities as control possibilities in parallel with the BCI. Sometimes these people have residual activity of their muscles. Therefore, in the presented hybrid BCI framework we want to explore the parallel usage of electroencephalographic (EEG) and electromyographic (EMG) activity, whereby the control abilities of both channels are fused. Results showed that the participants could achieve a good control of their hybrid BCI independently of their level of muscular fatigue. Thereby the multimodal fusion approach of muscular and brain activity yielded better and more stable performance compared to the single conditions. Even in the case of an increasing muscular fatigue a good control (moderate and graceful degradation of the performance compared to the non-fatigued case) and a smooth handover could be achieved. Therefore, such systems allow the users a very reliable hybrid BCI control although they are getting more and more exhausted or fatigued during the day.

Multimodal fusion of muscle and brain signals for a hybrid-BCI

Practical Brain-Computer Interfaces (BCIs) for disabled people should allow them to use all their remaining functionalities as control possibilities. Sometimes these people have residual activity of their muscles, most likely in the morning when they are not exhausted. In this work we fuse electromyographic (EMG) with electroencephalographic (EEG) activity in the framework of a so called "Hybrid-BCI" (hBCI) approach. Thereby, subjects could achieve a good control of their hBCI independently of their level of muscular fatigue. Furthermore, although EMG alone yields good performance, it is outperformed by the hybrid fusing of EEG and EMG. Two different fusion techniques are explored showing graceful performance degradation in the case of signal attenuation. Such a system allows a very reliable control and a smooth handover if the subjects get exhausted or fatigued during the day.

A concept to standardize raw biosignal transmission for brain-computer interfaces

With this concept we introduced the attempt of a standardized interface called TiA to transmit raw biosignals. TiA is able to deal with multirate and block-oriented data transmission. Data is distinguished by different signal types (e.g., EEG, EOG, NIRS, …), whereby those signals can be acquired at the same time from different acquisition devices. TiA is built as a client-server model. Multiple clients can connect to one server. Information is exchanged via a control- and a separated data connection. Control commands and meta information are transmitted over the control connection. Raw biosignal data is delivered using the data connection in a unidirectional way. For this purpose a standardized handshaking protocol and raw data packet have been developed. Thus, an abstraction layer between hardware devices and data processing was evolved facilitating standardization.

Combining Brain-Computer Interfaces and Assistive Technologies: State-of-the-Art and Challenges

In recent years, new research has brought the field of electroencephalogram (EEG)-based brain–computer interfacing (BCI) out of its infancy and into a phase of relative maturity through many demonstrated prototypes such as brain-controlled wheelchairs, keyboards, and computer games. With this proof-of-concept phase in the past, the time is now ripe to focus on the development of practical BCI technologies that can be brought out of the lab and into real-world applications. In particular, we focus on the prospect of improving the lives of countless disabled individuals through a combination of BCI technology with existing assistive technologies (AT). In pursuit of more practical BCIs for use outside of the lab, in this paper, we identify four application areas where disabled individuals could greatly benefit from advancements in BCI technology, namely, “Communication and Control”, “Motor Substitution”, “Entertainment”, and “Motor Recovery”. We review the current state of the art and possible future developments, while discussing the main research issues in these four areas. In particular, we expect the most progress in the development of technologies such as hybrid BCI architectures, user–machine adaptation algorithms, the exploitation of users’ mental states for BCI reliability and confidence measures, the incorporation of principles in human–computer interaction (HCI) to improve BCI usability, and the development of novel BCI technology including better EEG devices.

The hybrid BCI

Nowadays, everybody knows what a hybrid car is. A hybrid car normally has two engines to enhance energy efficiency and reduce CO2 output. Similarly, a hybrid brain-computer interface (BCI) is composed of two BCIs, or at least one BCI and another system. A hybrid BCI, like any BCI, must fulfill the following four criteria: (i) the device must rely on signals recorded directly from the brain; (ii) there must be at least one recordable brain signal that the user can intentionally modulate to effect goal-directed behaviour; (iii) real time processing; and (iv) the user must obtain feedback. This paper introduces hybrid BCIs that have already been published or are in development. We also introduce concepts for future work. We describe BCIs that classify two EEG patterns: one is the event-related (de)synchronisation (ERD, ERS) of sensorimotor rhythms, and the other is the steady-state visual evoked potential (SSVEP). Hybrid BCIs can either process their inputs simultaneously, or operate two systems sequentially, where the first system can act as a “brain switch”. For example, we describe a hybrid BCI that simultaneously combines ERD and SSVEP BCIs. We also describe a sequential hybrid BCI, in which subjects could use a brain switch to control an SSVEP-based hand orthosis. Subjects who used this hybrid BCI exhibited about half the false positives encountered while using the SSVEP BCI alone. A brain switch can also rely on hemodynamic changes measured through near-infrared spectroscopy (NIRS). Hybrid BCIs can also use one brain signal and a different type of input. This additional input can be an electrophysiological signal such as the heart rate, or a signal from an external device such as an eye tracking system.

Toward a hybrid brain-computer interface based on imagined movement and visual attention

Brain-computer interface (BCI) systems do not work for all users. This article introduces a novel combination of tasks that could inspire BCI systems that are more accurate than conventional BCIs, especially for users who cannot attain accuracy adequate for effective communication. Subjects performed tasks typically used in two BCI approaches, namely event-related desynchronization (ERD) and steady state visual evoked potential (SSVEP), both individually and in a 'hybrid' condition that combines both tasks. Electroencephalographic (EEG) data were recorded across three conditions. Subjects imagined moving the left or right hand (ERD), focused on one of the two oscillating visual stimuli (SSVEP), and then simultaneously performed both tasks. Accuracy and subjective measures were assessed. Offline analyses suggested that half of the subjects did not produce brain patterns that could be accurately discriminated in response to at least one of the two tasks. If these subjects produced comparable EEG patterns when trying to use a BCI, these subjects would not be able to communicate effectively because the BCI would make too many errors. Results also showed that switching to a different task used in BCIs could improve accuracy in some of these users. Switching to a hybrid approach eliminated this problem completely, and subjects generally did not consider the hybrid condition more difficult. Results validate this hybrid approach and suggest that subjects who cannot use a BCI should consider switching to a different BCI approach, especially a hybrid BCI. Subjects proficient with both approaches might combine them to increase information throughput by improving accuracy, reducing selection time, and/or increasing the number of possible commands.

Improved signal processing approaches in an offline simulation of a hybrid brain-computer interface

In a conventional brain-computer interface (BCI) system, users perform mental tasks that yield specific patterns of brain activity. A pattern recognition system determines which brain activity pattern a user is producing and thereby infers the user's mental task, allowing users to send messages or commands through brain activity alone. Unfortunately, despite extensive research to improve classification accuracy, BCIs almost always exhibit errors, which are sometimes so severe that effective communication is impossible. We recently introduced a new idea to improve accuracy, especially for users with poor performance. In an offline simulation of a "hybrid" BCI, subjects performed two mental tasks independently and then simultaneously. This hybrid BCI could use two different types of brain signals common in BCIs - event-related desynchronization (ERD) and steady-state evoked potentials (SSEPs). This study suggested that such a hybrid BCI is feasible. Here, we re-analyzed the data from our initial study. We explored eight different signal processing methods that aimed to improve classification and further assess both the causes and the extent of the benefits of the hybrid condition. Most analyses showed that the improved methods described here yielded a statistically significant improvement over our initial study. Some of these improvements could be relevant to conventional BCIs as well. Moreover, the number of illiterates could be reduced with the hybrid condition. Results are also discussed in terms of dual task interference and relevance to protocol design in hybrid BCIs.

A survey of stimulation methods used in SSVEP-based BCIs

Brain-computer interface (BCI) systems based on the steady-state visual evoked potential (SSVEP) provide higher information throughput and require shorter training than BCI systems using other brain signals. To elicit an SSVEP, a repetitive visual stimulus (RVS) has to be presented to the user. The RVS can be rendered on a computer screen by alternating graphical patterns, or with external light sources able to emit modulated light. The properties of an RVS (e.g., frequency, color) depend on the rendering device and influence the SSVEP characteristics. This affects the BCI information throughput and the levels of user safety and comfort. Literature on SSVEP-based BCIs does not generally provide reasons for the selection of the used rendering devices or RVS properties. In this paper, we review the literature on SSVEP-based BCIs and comprehensively report on the different RVS choices in terms of rendering devices, properties, and their potential influence on BCI performance, user safety and comfort.

Flexibility and practicality graz brain-computer interface approach

"Graz brain-computer interface (BCI)" transforms changes in oscillatory electroencephalogram (EEG) activity into control signals for external devices and feedback. Steady-state evoked potentials (SSEPs) and event-related desynchronization (ERD) are employed to encode user messages. User-specific setup and training are important issues for robust and reliable classification. Furthermore, in order to implement small and thus affordable systems, focus is put on the minimization of the number of EEG sensors. The system also supports the self-paced operation mode, that is, users have on-demand access to the system at any time and can autonomously initiate communication. Flexibility, usability, and practicality are essential to increase user acceptance. Here, we illustrate the possibilities offered by now from EEG-based communication. Results of several studies with able-bodied and disabled individuals performed inside the laboratory and in real-world environments are presented; their characteristics are shown and open issues are mentioned. The applications include the control of neuroprostheses and spelling devices, the interaction with Virtual Reality, and the operation of off-the-shelf software such as Google Earth.