A multi-signature brain-computer interface: use of transient and steady-state responses

EEG-based brain-computer interfaces exploiting steady-state somatosensory-evoked potentials: a literature review

A brain-computer interface (BCI) aims to derive commands from the user's brain activity in order to relay them to an external device. To do so, it can either detect a spontaneous change in the mental state, in the so-called 'active' BCIs, or a transient or sustained change in the brain response to an external stimulation, in 'reactive' BCIs. In the latter, external stimuli are perceived by the user through a sensory channel, usually sight or hearing. When the stimulation is sustained and periodical, the brain response reaches an oscillatory steady-state that can be detected rather easily. We focus our attention on electroencephalography-based BCIs (EEG-based BCI) in which a periodical signal, either mechanical or electrical, stimulates the user skin. This type of stimulus elicits a steady-state response of the somatosensory system that can be detected in the recorded EEG. The oscillatory and phase-locked voltage component characterising this response is called a steady-state somatosensory-evoked potential (SSSEP). It has been shown that the amplitude of the SSSEP is modulated by specific mental tasks, for instance when the user focuses their attention or not to the somatosensory stimulation, allowing the translation of this variation into a command. Actually, SSSEP-based BCIs may benefit from straightforward analysis techniques of EEG signals, like reactive BCIs, while allowing self-paced interaction, like active BCIs. In this paper, we present a survey of scientific literature related to EEG-based BCI exploiting SSSEP. Firstly, we endeavour to describe the main characteristics of SSSEPs and the calibration techniques that allow the tuning of stimulation in order to maximise their amplitude. Secondly, we present the signal processing and data classification algorithms implemented by authors in order to elaborate commands in their SSSEP-based BCIs, as well as the classification performance that they evaluated on user experiments.

Wheelchair Control in a Virtual Environment by Healthy Participants Using a P300-BCI Based on Tactile Stimulation: Training Effects and Usability

Tactile stimulation is less frequently used than visual for brain-computer interface (BCI) control, partly because of limitations in speed and accuracy. Non-visual BCI paradigms, however, may be required for patients who struggle with vision dependent BCIs because of a loss of gaze control. With the present study, we attempted to replicate earlier results by Herweg et al. (2016), with several minor adjustments and a focus on training effects and usability. We invited 16 healthy participants and trained them with a 4-class tactile P300-based BCI in five sessions. Their main task was to navigate a virtual wheelchair through a 3D apartment using the BCI. We found significant training effects on information transfer rate (ITR), which increased from a mean of 3.10–9.50 bits/min. Further, both online and offline accuracies significantly increased with training from 65% to 86% and 70% to 95%, respectively. We found only a descriptive increase of P300 amplitudes at Fz and Cz with training. Furthermore, we report subjective data from questionnaires, which indicated a relatively high workload and moderate to high satisfaction. Although our participants have not achieved the same high performance as in the Herweg et al. (2016) study, we provide evidence for training effects on performance with a tactile BCI and confirm the feasibility of the paradigm.

A Survey on the Use of Haptic Feedback for Brain-Computer Interfaces and Neurofeedback

Neurofeedback (NF) and brain-computer interface (BCI) applications rely on the registration and real-time feedback of individual patterns of brain activity with the aim of achieving self-regulation of specific neural substrates or control of external devices. These approaches have historically employed visual stimuli. However, in some cases vision is unsuitable or inadequately engaging. Other sensory modalities, such as auditory or haptic feedback have been explored, and multisensory stimulation is expected to improve the quality of the interaction loop. Moreover, for motor imagery tasks, closing the sensorimotor loop through haptic feedback may be relevant for motor rehabilitation applications, as it can promote plasticity mechanisms. This survey reviews the various haptic technologies and describes their application to BCIs and NF. We identify major trends in the use of haptic interfaces for BCI and NF systems and discuss crucial aspects that could motivate further studies.

Recent Advances in Hybrid Brain-Computer Interface Systems: A Technological and Quantitative Review

Brain-Computer Interface (BCI) is a system that enables users to transmit commands to the computer using their brain activity recorded by electroencephalography. In a Hybrid Brain-Computer Interface (HBCI), a BCI control signal combines with one or more BCI control signals or with Human-Machine Interface (HMI) biosignals to increase classification accuracy, boost system speed, and improve user’s satisfaction. HBCI systems are categorized according to the type of combined signals and the combination technique (simultaneous or sequential). They have been used in several applications such as cursor control, target selection, and spellers. Increasing the number of articles published in this field indicates the significance of these systems. In this paper, different HBCI combinations, their important features, and potential applications are discussed. In most cases, the combination of a BCI control signal with a HMI biosignal yields higher information transfer rate than two BCI control signals.

Improvement in Classification of Tactile Event-Related Potentials using Random-Interval Tasks∗

The classification accuracy of an event-related potential (ERP) induced by a tactile stimulus is investigated to create a brain-computer interface (BCI). Mechanical tactile stimuli were applied to the left and right index fingers using two piezo actuator probes that were used as a Braille display for the visually impaired. In an experiment, two-class classification was investigated using three kinds of tactile stimulus pattern. The subjects were instructed to pay attention to unusual target stimuli while avoiding other frequent nontarget stimuli. The extracted features were classified using stepwise linear discriminant analysis. As a result, high accuracy was obtained by the task of random intervals compared with the task of constant intervals. It was suggested that the accuracy of the BCI using tactile stimuli is influenced by the concentration on the task.

Fusion of Facial Expressions and EEG for Multimodal Emotion Recognition

This paper proposes two multimodal fusion methods between brain and peripheral signals for emotion recognition. The input signals are electroencephalogram and facial expression. The stimuli are based on a subset of movie clips that correspond to four specific areas of valance-arousal emotional space (happiness, neutral, sadness, and fear). For facial expression detection, four basic emotion states (happiness, neutral, sadness, and fear) are detected by a neural network classifier. For EEG detection, four basic emotion states and three emotion intensity levels (strong, ordinary, and weak) are detected by two support vector machines (SVM) classifiers, respectively. Emotion recognition is based on two decision-level fusion methods of both EEG and facial expression detections by using a sum rule or a production rule. Twenty healthy subjects attended two experiments. The results show that the accuracies of two multimodal fusion detections are 81.25% and 82.75%, respectively, which are both higher than that of facial expression (74.38%) or EEG detection (66.88%). The combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources.

A systematic review of hybrid brain-computer interfaces: Taxonomy and usability perspectives

A new Brain-Computer Interface (BCI) technique, which is called a hybrid BCI, has recently been proposed to address the limitations of conventional single BCI system. Although some hybrid BCI studies have shown promising results, the field of hybrid BCI is still in its infancy and there is much to be done. Especially, since the hybrid BCI systems are so complicated and complex, it is difficult to understand the constituent and role of a hybrid BCI system at a glance. Also, the complicated and complex systems make it difficult to evaluate the usability of the systems. We systematically reviewed and analyzed the current state-of-the-art hybrid BCI studies, and proposed a systematic taxonomy for classifying the types of hybrid BCIs with multiple taxonomic criteria. After reviewing 74 journal articles, hybrid BCIs could be categorized with respect to 1) the source of brain signals, 2) the characteristics of the brain signal, and 3) the characteristics of operation in each system. In addition, we exhaustively reviewed recent literature on usability of BCIs. To identify the key evaluation dimensions of usability, we focused on task and measurement characteristics of BCI usability. We classified and summarized 31 BCI usability journal articles according to task characteristics (type and description of task) and measurement characteristics (subjective and objective measures). Afterwards, we proposed usability dimensions for BCI and hybrid BCI systems according to three core-constructs: Satisfaction, effectiveness, and efficiency with recommendations for further research. This paper can help BCI researchers, even those who are new to the field, can easily understand the complex structure of the hybrid systems at a glance. Recommendations for future research can also be helpful in establishing research directions and gaining insight in how to solve ergonomics and HCI design issues surrounding BCI and hybrid BCI systems by usability evaluation.

Quantification of task-dependent cortical activation evoked by robotic continuous wrist joint manipulation in chronic hemiparetic stroke

Background

Cortical damage after stroke can drastically impair sensory and motor function of the upper limb, affecting the execution of activities of daily living and quality of life. Motor impairment after stroke has been thoroughly studied, however sensory impairment and its relation to movement control has received less attention. Integrity of the somatosensory system is essential for feedback control of human movement, and compromised integrity due to stroke has been linked to sensory impairment.

Methods

The goal of this study is to assess the integrity of the somatosensory system in individuals with chronic hemiparetic stroke with different levels of sensory impairment, through a combination of robotic joint manipulation and high-density electroencephalogram (EEG). A robotic wrist manipulator applied continuous periodic disturbances to the affected limb, providing somatosensory (proprioceptive and tactile) stimulation while challenging task execution. The integrity of the somatosensory system was evaluated during passive and active tasks, defined as ‘relaxed wrist’ and ‘maintaining 20% maximum wrist flexion’, respectively. The evoked cortical responses in the EEG were quantified using the power in the averaged responses and their signal-to-noise ratio.

Results

Thirty individuals with chronic hemiparetic stroke and ten unimpaired individuals without stroke participated in this study. Participants with stroke were classified as having severe, mild, or no sensory impairment, based on the Erasmus modification of the Nottingham Sensory Assessment. Under passive conditions, wrist manipulation resulted in contralateral cortical responses in unimpaired and chronic stroke participants with mild and no sensory impairment. In participants with severe sensory impairment the cortical responses were strongly reduced in amplitude, which related to anatomical damage. Under active conditions, participants with mild sensory impairment showed reduced responses compared to the passive condition, whereas unimpaired and chronic stroke participants without sensory impairment did not show this reduction.

Conclusions

Robotic continuous joint manipulation allows studying somatosensory cortical evoked responses during the execution of meaningful upper limb control tasks. Using such an approach it is possible to quantitatively assess the integrity of sensory pathways; in the context of movement control this provides additional information required to develop more effective neurorehabilitation therapies.

A hybrid three-class brain-computer interface system utilizing SSSEPs and transient ERPs

OBJECTIVE

This paper investigates the fusion of steady-state somatosensory evoked potentials (SSSEPs) and transient event-related potentials (tERPs), evoked through tactile simulation on the left and right-hand fingertips, in a three-class EEG based hybrid brain-computer interface. It was hypothesized, that fusing the input signals leads to higher classification rates than classifying tERP and SSSEP individually.

APPROACH

Fourteen subjects participated in the studies, consisting of a screening paradigm to determine person dependent resonance-like frequencies and a subsequent online paradigm. The whole setup of the BCI system was based on open interfaces, following suggestions for a common implementation platform. During the online experiment, subjects were instructed to focus their attention on the stimulated fingertips as indicated by a visual cue. The recorded data were classified during runtime using a multi-class shrinkage LDA classifier and the outputs were fused together applying a posterior probability based fusion. Data were further analyzed offline, involving a combined classification of SSSEP and tERP features as a second fusion principle. The final results were tested for statistical significance applying a repeated measures ANOVA.

MAIN RESULTS

A significant classification increase was achieved when fusing the results with a combined classification compared to performing an individual classification. Furthermore, the SSSEP classifier was significantly better in detecting a non-control state, whereas the tERP classifier was significantly better in detecting control states. Subjects who had a higher relative band power increase during the screening session also achieved significantly higher classification results than subjects with lower relative band power increase.

SIGNIFICANCE

It could be shown that utilizing SSSEP and tERP for hBCIs increases the classification accuracy and also that tERP and SSSEP are not classifying control- and non-control states with the same level of accuracy.

Commanding a Brain-Controlled Wheelchair Using Steady-State Somatosensory Evoked Potentials

In this work, we propose a novel brain-controlled wheelchair, one of the major applications of brain-machine interfaces (BMIs), that allows an individual with mobility impairments to perform daily living activities independently. Specifically, we propose to use a steady-state somatosensory evoked potential (SSSEP) paradigm, which elicits brain responses to tactile stimulation of specific frequencies, for a user's intention to control a wheelchair. In our system, a user had three possible commands by concentrating on one of three vibration stimuli, which were attached to the left-hand, right-hand, and right-foot, to selectively control the wheelchair. The three stimuli were associated with three wheelchair commands: turn-left, turn-right, and move-forward. From a machine learning perspective, we also devise a novel feature representation by combining spatial and spectral characteristics of brain signals. In order to validate the effectiveness of the proposed SSSEP-based system, we considered two different tasks: 1) a simple obstacle-avoidance task within a limited time and; 2) a driving task along the predefined trajectory of about 40 m length, where there were a narrow pathway, a door, and obstacles. In both experiments, we recruited 12 subjects and compared the average time of motor imagery (MI) and SSSEP-based controls to complete the task. With the SSSEP-based control, all subjects successfully completed the task without making any collision while four subjects failed it with MI-based control. It is also noteworthy that in terms of the average time to complete the task, the SSSEP-based control outperformed the MI-based control. In the other more challenging task, all subjects successfully reached the target location.

The Role of Transient Target Stimuli in a Steady-State Somatosensory Evoked Potential-Based Brain-Computer Interface Setup

In earlier literature, so-called twitches were used to support a user in a steady-state somatosensory evoked potential (SSSEP) based brain–computer interface (BCI) to focus attention on the requested targets. Within this work, we investigate the impact of these transient target stimuli on SSSEPs in a real-life BCI setup. A hybrid BCI was designed which combines SSSEPs and P300 potentials evoked by twitches randomly embedded into the streams of tactile stimuli. The EEG of fourteen healthy subjects was recorded, while their left and right index fingers were simultaneously stimulated using frequencies selected in a screening procedure. The subjects were randomly instructed by a cue on a screen to focus attention on one or none of the fingers. Feature for SSSEPs and P300 potentials were extracted and classified using separately trained multi-class shrinkage LDA classifiers. Three-class classification accuracies significantly better than random could be reached by nine subjects using SSSEP features and by 12 subjects using P300 features respectively. The average classification accuracies were 48.6% using SSSEP and 50.7% using P300 features. By means of a Monte Carlo permutation test it could be shown that twitches have an attenuation effect on the SSSEP. Significant SSSEP blocking effects time-locked to twitch positions were found in seven subjects. Our findings suggest that the attempt to combine different types of stimulation signals like repetitive signals and twitches has a mutual influence on each other, which may be the main reason for the rather moderate BCI performance. This influence is originated at the level of stimulus generation but becomes apparent as physiological effect in the SSSEP. When designing a hybrid BCI based on SSSEPs and P300 potentials, one has to find an optimal tradeoff depending on the overall design goals or individual subjects' performance. Our results give therefore some new insights that may be useful for the successful design of hybrid BCIs.

Connections between intraparietal sulcus and a sensorimotor network underpin sustained tactile attention

Previous studies on sustained tactile attention draw conclusions about underlying cortical networks by averaging over experimental conditions without considering attentional variance in single trials. This may have formed an imprecise picture of brain processes underpinning sustained tactile attention. In the present study, we simultaneously recorded EEG-fMRI and used modulations of steady-state somatosensory evoked potentials (SSSEPs) as a measure of attentional trial-by-trial variability. Therefore, frequency-tagged streams of vibrotactile stimulations were simultaneously presented to both index fingers. Human participants were cued to sustain attention to either the left or right finger stimulation and to press a button whenever they perceived a target pulse embedded in the to-be-attended stream. In-line with previous studies, a classical general linear model (GLM) analysis based on cued attention conditions revealed increased activity mainly in somatosensory and cerebellar regions. Yet, parametric modeling of the BOLD response using simultaneously recorded SSSEPs as a marker of attentional trial-by-trial variability quarried the intraparietal sulcus (IPS). The IPS in turn showed enhanced functional connectivity to a modality-unspecific attention network. However, this was only revealed on the basis of cued attention conditions in the classical GLM. By considering attentional variability as captured by SSSEPs, the IPS showed increased connectivity to a sensorimotor network, underpinning attentional selection processes between competing tactile stimuli and action choices (press a button or not). Thus, the current findings highlight the potential value by considering attentional variations in single trials and extend previous knowledge on the role of the IPS in tactile attention.

Combining and comparing EEG, peripheral physiology and eye-related measures for the assessment of mental workload

While studies exist that compare different physiological variables with respect to their association with mental workload, it is still largely unclear which variables supply the best information about momentary workload of an individual and what is the benefit of combining them. We investigated workload using the n-back task, controlling for body movements and visual input. We recorded EEG, skin conductance, respiration, ECG, pupil size and eye blinks of 14 subjects. Various variables were extracted from these recordings and used as features in individually tuned classification models. Online classification was simulated by using the first part of the data as training set and the last part of the data for testing the models. The results indicate that EEG performs best, followed by eye related measures and peripheral physiology. Combining variables from different sensors did not significantly improve workload assessment over the best performing sensor alone. Best classification accuracy, a little over 90%, was reached for distinguishing between high and low workload on the basis of 2 min segments of EEG and eye related variables. A similar and not significantly different performance of 86% was reached using only EEG from single electrode location Pz.