Control of a 7-DOF Robotic Arm System With an SSVEP-Based BCI

Trajectory Decoding of Arm Reaching Movement Imageries for Brain-Controlled Robot Arm System

Development of noninvasive brain-machine interface (BMI) systems based on electroencephalography (EEG), driven by spontaneous movement intentions, is a useful tool for controlling external devices or supporting a neuro- rehabilitation. In this study, we present the possibility of brain-controlled robot arm system using arm trajectory decoding. To do that, we first constructed the experimental system that can acquire the EEG data for not only movement execution (ME) task but also movement imagery (MI) tasks. Five subjects participated in our experiments and performed four directional reaching tasks (Left, right, forward, and backward) in the 3D plane. For robust arm trajectory decoding, we propose a subject-dependent deep neural network (DNN) architecture. The decoding model applies the principle of bi-directional long short-term memory (LSTM) network. As a result, we confirmed the decoding performance (r-value: >0.8) for all X-, Y-, and Z-axis across all subjects in the MI as well as ME tasks. These results show the feasibility of the EEG-based intuitive robot arm control system for high-level tasks (e.g., drink water or moving some objects). Also, we confirm that the proposed method has no much decoding performance variations between ME and MI tasks for the offline analysis. Hence, we will demonstrate that the decoding model is capable of robust trajectory decoding even in a real-time environment.

Exploration of User's Mental State Changes during Performing Brain-Computer Interface

Substantial developments have been established in the past few years for enhancing the performance of brain–computer interface (BCI) based on steady-state visual evoked potential (SSVEP). The past SSVEP-BCI studies utilized different target frequencies with flashing stimuli in many different applications. However, it is not easy to recognize user’s mental state changes when performing the SSVEP-BCI task. What we could observe was the increasing EEG power of the target frequency from the user’s visual area. BCI user’s cognitive state changes, especially in mental focus state or lost-in-thought state, will affect the BCI performance in sustained usage of SSVEP. Therefore, how to differentiate BCI users’ physiological state through exploring their neural activities changes while performing SSVEP is a key technology for enhancing the BCI performance. In this study, we designed a new BCI experiment which combined working memory task into the flashing targets of SSVEP task using 12 Hz or 30 Hz frequencies. Through exploring the EEG activity changes corresponding to the working memory and SSVEP task performance, we can recognize if the user’s cognitive state is in mental focus or lost-in-thought. Experiment results show that the delta (1–4 Hz), theta (4–7 Hz), and beta (13–30 Hz) EEG activities increased more in mental focus than in lost-in-thought state at the frontal lobe. In addition, the powers of the delta (1–4 Hz), alpha (8–12 Hz), and beta (13–30 Hz) bands increased more in mental focus in comparison with the lost-in-thought state at the occipital lobe. In addition, the average classification performance across subjects for the KNN and the Bayesian network classifiers were observed as 77% to 80%. These results show how mental state changes affect the performance of BCI users. In this work, we developed a new scenario to recognize the user’s cognitive state during performing BCI tasks. These findings can be used as the novel neural markers in future BCI developments.

Validation of a brain-computer interface version of the digit symbol substitution test in healthy subjects

Digit symbol substitution test (DSST), which is a valid and sensitive tool to assess human cognitive dysfunction, has been widely used in clinical neuropsychology. Although several versions of DSST are currently available, most of the existing DSST versions rely on examinees' intact motor function. This limits their utility in severely motor-impaired individuals. A brain-computer interface (BCI) version of DSST was implemented in this study. Steady-state visual evoked potential (SSVEP) was adopted to build the BCI. Nine symbols in the proposed SSVEP BCI-based DSST were designed with clearly different shapes for decreasing measurement errors due to misidentified symbols. To reduce practice effect, furthermore, the digit-symbol pairs of each trial were different. A two-target SSVEP BCI was designed to judge whether the digit-symbol probe in the center of the user interface matched one of the nine digit-symbol pairs above the user interface. All 12 examinees were able to perform the tasks using the proposed SSVEP BCI-based DSST with 96.17 ± 4.18% averaged accuracy, which was comparable with that of computerized DSST. Furthermore, for examinees participating in both offline and online experiment, the accuracies of the online and offline experiments were comparable, supporting that the proposed BCI-DSST was reliable for repeatedly evaluating examinees' cognitive function over time. These results verified that the proposed SSVEP BCI-based DSST was feasible and effective for healthy subjects.

An online SSVEP-BCI system in an optical see-through augmented reality environment

OBJECTIVE

This study aimed to design and evaluate a high-speed online steady-state visually evoked potential (SSVEP)-based brain-computer interface (BCI) in an optical see-through (OST) augmented reality (AR) environment.

APPROACH

An eight-class BCI was designed in an OST-AR headset which is wearable and allows users to see the user interface of the BCI and the device to be controlled in the same view field via the OST head-mounted display. The accuracies, information transfer rates (ITRs), and SSVEP signal characteristics of the AR-BCI were evaluated and compared with a computer screen-based BCI implemented with a laptop in offline and online cue-guided tasks. Then, the performance of the AR-BCI was evaluated in an online robotic arm control task.

MAIN RESULTS

The offline results obtained during the cue-guided task performed with the AR-BCI showed maximum averaged ITRs of 65.50  ±  9.86 bits min^-1 according to the extended canonical correlation analysis-based target identification method. The online cue-guided task achieved averaged ITRs of 65.03  ±  11.40 bits min^-1. The online robotic arm control task achieved averaged ITRs of 45.57  ±  7.40 bits min^-1. Compared with the screen-based BCI, some limitations of the AR environment impaired BCI performance and the quality of SSVEP signals.

SIGNIFICANCE

The results showed the potential for providing a high-performance brain-control interaction method by combining AR and BCI. This study could provide methodological guidelines for developing more wearable BCIs in OST-AR environments and will also encourage more interesting applications involving BCIs and AR techniques.

Effects of stimulation frequency and stimulation waveform on steady-state visual evoked potentials using a computer monitor

OBJECTIVE

A visual stimulator plays a vital part in brain-computer interfaces (BCIs) based on steady-state visual evoked potential (SSVEP). The properties of visual stimulation, such as frequency, color, and waveform, will influence SSVEP-based BCI performance to some extent. Recently, the computer monitor serves as a visual stimulator that is widespread in SSVEP-based BCIs because of its great flexibility in generating visual stimuli. However, stimulation properties based on a computer monitor have received very little attention. For a better comprehension of SSVEPs, this study explored the stimulation effects of waveforms and frequencies, when evoking SSVEPs through a computer monitor.

APPROACH

This study utilized the approximation methods to realize sine- and square-wave temporal modulations at 18 stimulation frequencies ranging from 6 to 40 Hz on a conventional 120 Hz LCD screen. We collected electroencephalogram (EEG) datasets from 12 healthy subjects and compared the signal-to-noise ratios (SNRs), amplitudes, and topographic mapping of SSVEPs evoked by these two temporal modulation flickers (sine- and square-wave). In addition, a BCI experiment with two nine-target BCIs (i.e. low-frequency BCI and high-frequency BCI) was implemented to compare the two stimulation waveforms in terms of BCI performance.

MAIN RESULTS

For both sine- and square-wave stimulation conditions, strong SSVEPs over the occipital area were observed for each stimulation frequency. SSVEP amplitudes at the stimulation frequency exhibited a global peak in the low-frequency band. The second harmonic SSVEP frequency-response functions showed the largest amplitude at 6 Hz and fell sharply for higher frequencies. In the BCI experiment, the classification performance of the square-wave stimuli was notably higher than that of the sine-wave stimuli when using shorter data lengths.

SIGNIFICANCE

These results suggested that the square-wave flicker was more efficient at implementing high-speed BCIs based on SSVEP when using a computer monitor as a visual stimulator.

A Method for Suppressing Electrical Stimulation Artifacts from Electromyography

When surface electromyography (EMG) signal is used in a real-time functional electrical stimulation (FES) system for feedback control, the artifact from electrical stimulation is a key challenge for EMG signal processing. To address this challenge, this study proposes a novel method to suppress stimulation artifacts in the EMG-driven closed-loop FES system. The proposed method is inspired by an experimental study that compares artifacts generated by electrical stimulations with different current intensities. It is found that (1) spikes of stimulation artifacts are susceptible to the current intensity and (2) tailing components are similar under different current intensities. Based on these observations, the proposed method combines the blanking and template subtracting strategies for suppressing stimulation artifact. The length of blanking window for suppressing the stimulation spike is adaptively determined by a spike detection algorithm and the first-order derivative analysis of signal. An autoregressive model is used to estimate the tailing part of stimulation artifact, which is an adaptive template for subtracting the artifact. The proposed method is evaluated on both semi-synthetic and experimental datasets. Verified on the semi-synthetic dataset, the proposed method achieves better performance than the classic blanking method. Validated on the experimental dataset, the proposed method substantially decreases the power of stimulation artifact in the EMG. These results indicate that the proposed method can effectively suppress the stimulation artifact while retains the useful EMG signal for an EMG-driven FES system.

Characterized Bioelectric Signals by Means of Neural Networks and Wavelets to Remotely Control a Human-Machine Interface

Everyday, people interact with different types of human machine interfaces, and the use of them is increasing, thus, it is necessary to design interfaces which are capable of responding in an intelligent, natural, inexpensive, and accessible way, regardless of social, cultural, economic, or physical features of a user. In this sense, it has been sought out the development of small interfaces to avoid any type of user annoyance. In this paper, bioelectric signals have been analyzed and characterized in order to propose a more natural human-machine interaction system. The proposed scheme is controlled by electromyographic signals that a person can create through arm movements. Such arm signals have been analyzed and characterized by a back-propagation neural network, and by a wavelet analysis, in this way control commands were obtained from such arm electromyographic signals. The developed interface, uses Extensible Messaging and Presence Protocol (XMPP) to send control commands remotely. In the experiment, it manipulated a vehicle that was approximately 52 km away from the user, with which it can be showed that a characterized electromyographic signal can be sufficient for controlling embedded devices such as a Raspberri Pi, and in this way we can use the neural network and the wavelet analysis to generate control words which can be used inside the Internet of Things too. A Tiva-C board has been used to acquire data instead of more popular development boards, with an adequate response. One of the most important aspects related to the proposed interface is that it can be used by almost anyone, including people with different abilities and even illiterate people. Due to the existence of individual efforts to characterize different types of bioelectric signals, we propose the generation of free access Bioelectric Control Dictionary, to define and consult each characterized biosignal.