Control of a 7-DOF Robotic Arm System With an SSVEP-Based BCI

Visual tracking brain-computer interface

Brain-computer interfaces (BCIs) offer a way to interact with computers without relying on physical movements. Non-invasive electroencephalography-based visual BCIs, known for efficient speed and calibration ease, face limitations in continuous tasks due to discrete stimulus design and decoding methods. To achieve continuous control, we implemented a novel spatial encoding stimulus paradigm and devised a corresponding projection method to enable continuous modulation of decoded velocity. Subsequently, we conducted experiments involving 17 participants and achieved Fitt's information transfer rate (ITR) of 0.55 bps for the fixed tracking task and 0.37 bps for the random tracking task. The proposed BCI with a high Fitt's ITR was then integrated into two applications, including painting and gaming. In conclusion, this study proposed a visual BCI based-control method to go beyond discrete commands, allowing natural continuous control based on neural activity.

Applying Common Spatial Pattern and Convolutional Neural Network to Classify Movements via EEG Signals

Developing an electroencephalography (EEG)-based brain-computer interface (BCI) system is crucial to enhancing the control of external prostheses by accurately distinguishing various movements through brain signals. This innovation can provide comfortable circumstances for the populace who have movement disabilities. This study combined the most prospering methods used in BCI systems, including one-versus-rest common spatial pattern (OVR-CSP) and convolutional neural network (CNN), to automatically extract features and classify eight different movements of the shoulder, wrist, and elbow via EEG signals. The number of subjects who participated in the experiment was 10, and their EEG signals were recorded while performing movements at fast and slow speeds. We used preprocessing techniques before transforming EEG signals into another space by OVR-CSP, followed by sending signals into the CNN architecture consisting of four convolutional layers. Moreover, we extracted feature vectors after applying OVR-CSP and considered them as inputs to KNN, SVM, and MLP classifiers. Then, the performance of these classifiers was compared with the CNN method. The results demonstrated that the classification of eight movements using the proposed CNN architecture obtained an average accuracy of 97.65% for slow movements and 96.25% for fast movements in the subject-independent model. This method outperformed other classifiers with a substantial difference; ergo, it can be useful in improving BCI systems for better control of prostheses.

Unsupervised Neural Manifold Alignment for Stable Decoding of Movement from Cortical Signals

The stable decoding of movement parameters using neural activity is crucial for the success of brain-machine interfaces (BMIs). However, neural activity can be unstable over time, leading to changes in the parameters used for decoding movement, which can hinder accurate movement decoding. To tackle this issue, one approach is to transfer neural activity to a stable, low-dimensional manifold using dimensionality reduction techniques and align manifolds across sessions by maximizing correlations of the manifolds. However, the practical use of manifold stabilization techniques requires knowledge of the true subject intentions such as target direction or behavioral state. To overcome this limitation, an automatic unsupervised algorithm is proposed that determines movement target intention before manifold alignment in the presence of manifold rotation and scaling across sessions. This unsupervised algorithm is combined with a dimensionality reduction and alignment method to overcome decoder instabilities. The effectiveness of the BMI stabilizer method is represented by decoding the two-dimensional (2D) hand velocity of two rhesus macaque monkeys during a center-out-reaching movement task. The performance of the proposed method is evaluated using correlation coefficient and R-squared measures, demonstrating higher decoding performance compared to a state-of-the-art unsupervised BMI stabilizer. The results offer benefits for the automatic determination of movement intents in long-term BMI decoding. Overall, the proposed method offers a promising automatic solution for achieving stable and accurate movement decoding in BMI applications.

Advanced Electrode Technologies for Noninvasive Brain-Computer Interfaces

Brain-computer interfaces (BCIs) have garnered significant attention in recent years due to their potential applications in medical, assistive, and communication technologies. Building on this, noninvasive BCIs stand out as they provide a safe and user-friendly method for interacting with the human brain. In this work, we provide a comprehensive overview of the latest developments and advancements in material, design, and application of noninvasive BCIs electrode technology. We also explore the challenges and limitations currently faced by noninvasive BCI electrode technology and sketch out the technological roadmap from three dimensions: Materials and Design; Performances; Mode and Function. We aim to unite research efforts within the field of noninvasive BCI electrode technology, focusing on the consolidation of shared goals and fostering integrated development strategies among a diverse array of multidisciplinary researchers.

EEG-controlled tele-grasping for undefined objects

This paper presents a teleoperation system of robot grasping for undefined objects based on a real-time EEG (Electroencephalography) measurement and shared autonomy. When grasping an undefined object in an unstructured environment, real-time human decision is necessary since fully autonomous grasping may not handle uncertain situations. The proposed system allows involvement of a wide range of human decisions throughout the entire grasping procedure, including 3D movement of the gripper, selecting proper grasping posture, and adjusting the amount of grip force. These multiple decision-making procedures of the human operator have been implemented with six flickering blocks for steady-state visually evoked potentials (SSVEP) by dividing the grasping task into predefined substeps. Each substep consists of approaching the object, selecting posture and grip force, grasping, transporting to the desired position, and releasing. The graphical user interface (GUI) displays the current substep and simple symbols beside each flickering block for quick understanding. The tele-grasping of various objects by using real-time human decisions of selecting among four possible postures and three levels of grip force has been demonstrated. This system can be adapted to other sequential EEG-controlled teleoperation tasks that require complex human decisions.

Shared Three-Dimensional Robotic Arm Control Based on Asynchronous BCI and Computer Vision

OBJECTIVE

A brain-computer interface (BCI) can be used to translate neuronal activity into commands to control external devices. However, using noninvasive BCI to control a robotic arm for movements in three-dimensional (3D) environments and accomplish complicated daily tasks, such as grasping and drinking, remains a challenge.

APPROACH

In this study, a shared robotic arm control system based on hybrid asynchronous BCI and computer vision was presented. The BCI model, which combines steady-state visual evoked potentials (SSVEPs) and blink-related electrooculography (EOG) signals, allows users to freely choose from fifteen commands in an asynchronous mode corresponding to robot actions in a 3D workspace and reach targets with a wide movement range, while computer vision can identify objects and assist a robotic arm in completing more precise tasks, such as grasping a target automatically.

RESULTS

Ten subjects participated in the experiments and achieved an average accuracy of more than 92% and a high trajectory efficiency for robot movement. All subjects were able to perform the reach-grasp-drink tasks successfully using the proposed shared control method, with fewer error commands and shorter completion time than with direct BCI control.

SIGNIFICANCE

Our results demonstrated the feasibility and efficiency of generating practical multidimensional control of an intuitive robotic arm by merging hybrid asynchronous BCI and computer vision-based recognition.

BCI Control of a Robotic Arm Based on SSVEP With Moving Stimuli for Reach and Grasp Tasks

Brain-computer interface (BCI) provides a novel technology for patients and healthy human subjects to control a robotic arm. Currently, BCI control of a robotic arm to complete the reaching and grasping tasks in an unstructured environment is still challenging because the current BCI technology does not meet the requirement of manipulating a multi-degree robotic arm accurately and robustly. BCI based on steady-state visual evoked potential (SSVEP) could output a high information transfer rate; however, the conventional SSVEP paradigm failed to control a robotic arm to move continuously and accurately because the users have to switch their gaze between the flickering stimuli and the target frequently. This study proposed a novel SSVEP paradigm in which the flickering stimuli were attached to the robotic arm's gripper and moved with it. First, an offline experiment was designed to investigate the effects of moving flickering stimuli on the SSVEP's responses and decoding accuracy. After that, contrast experiments were conducted, and twelve subjects were recruited to participate in a robotic arm control experiment using both the paradigm one (P1, with moving flickering stimuli) and the paradigm two (P2, conventional fixed flickering stimuli) using a block randomization design to balance their sequences. Double blinks were used to trigger the grasping action asynchronously whenever the subjects were confident that the position of the robotic arm's gripper was accurate enough. Experimental results showed that the paradigm P1 with moving flickering stimuli provided a much better control performance than the conventional paradigm P2 in completing a reaching and grasping task in an unstructured environment. Subjects' subjective feedback scored by a NASA-TLX mental workload scale also corroborated the BCI control performance. The results of this study suggest that the proposed control interface based on SSVEP BCI provides a better solution for robotic arm control to complete the accurate reaching and grasping tasks.

Wireless EEG: A survey of systems and studies

The popular brain monitoring method of electroencephalography (EEG) has seen a surge in commercial attention in recent years, focusing mostly on hardware miniaturization. This has led to a varied landscape of portable EEG devices with wireless capability, allowing them to be used by relatively unconstrained users in real-life conditions outside of the laboratory. The wide availability and relative affordability of these devices provide a low entry threshold for newcomers to the field of EEG research. The large device variety and the at times opaque communication from their manufacturers, however, can make it difficult to obtain an overview of this hardware landscape. Similarly, given the breadth of existing (wireless) EEG knowledge and research, it can be challenging to get started with novel ideas. Therefore, this paper first provides a list of 48 wireless EEG devices along with a number of important-sometimes difficult-to-obtain-features and characteristics to enable their side-by-side comparison, along with a brief introduction to each of these aspects and how they may influence one's decision. Secondly, we have surveyed previous literature and focused on 110 high-impact journal publications making use of wireless EEG, which we categorized by application and analyzed for device used, number of channels, sample size, and participant mobility. Together, these provide a basis for informed decision making with respect to hardware and experimental precedents when considering new, wireless EEG devices and research. At the same time, this paper provides background material and commentary about pitfalls and caveats regarding this increasingly accessible line of research.

Control of a Production Manipulator with the Use of BCI in Conjunction with an Industrial PLC

Research in the field of gathering and analyzing biological signals is growing. The sensors are becoming more available and more non-invasive for examining such signals, which in the past required the inconvenient acquisition of data. This was achieved mainly by the fact that biological sensors were able to be built into wearable and portable devices. The representation and analysis of EEGs (electroencephalograms) is nowadays commonly used in various application areas. The application of the use of the EEG signals to the field of automation is still an unexplored area and therefore provides opportunities for interesting research. In our research, we focused on the area of processing automation; especially the use of the EEG signals to bridge the communication between control of individual processes and a human. In this study, the real-time communication between a PLC (programmable logic controller) and BCI (brain computer interface) was investigated and described. In the future, this approach can help people with physical disabilities to control certain machines or devices and therefore it could find applicability in overcoming physical disabilities. The main contribution of the article is, that we have demonstrated the possibility of interaction between a person and a manipulator controlled by a PLC with the help of a BCI. Potentially, with the expansion of functionality, such solutions will allow a person with physical disabilities to participate in the production process.

A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment

At present, single-modal brain-computer interface (BCI) still has limitations in practical application, such as low flexibility, poor autonomy, and easy fatigue for subjects. This study developed an asynchronous robotic arm control system based on steady-state visual evoked potentials (SSVEP) and eye-tracking in virtual reality (VR) environment, including simultaneous and sequential modes. For simultaneous mode, target classification was realized by decision-level fusion of electroencephalography (EEG) and eye-gaze. The stimulus duration for each subject was non-fixed, which was determined by an adjustable window method. Subjects could autonomously control the start and stop of the system using triple blink and eye closure, respectively. For sequential mode, no calibration was conducted before operation. First, subjects’ gaze area was obtained through eye-gaze, and then only few stimulus blocks began to flicker. Next, target classification was determined using EEG. Additionally, subjects could reject false triggering commands using eye closure. In this study, the system effectiveness was verified through offline experiment and online robotic-arm grasping experiment. Twenty subjects participated in offline experiment. For simultaneous mode, average ACC and ITR at the stimulus duration of 0.9 s were 90.50% and 60.02 bits/min, respectively. For sequential mode, average ACC and ITR at the stimulus duration of 1.4 s were 90.47% and 45.38 bits/min, respectively. Fifteen subjects successfully completed the online tasks of grabbing balls in both modes, and most subjects preferred the sequential mode. The proposed hybrid brain-computer interface (h-BCI) system could increase autonomy, reduce visual fatigue, meet individual needs, and improve the efficiency of the system.

Assembling global and local spatial-temporal filters to extract discriminant information of EEG in RSVP task

Objective.Brain-computer interface (BCI) system has developed rapidly in the past decade. And rapid serial visual presentation (RSVP) is an important BCI paradigm to detect the targets in high-speed image streams. For decoding electroencephalography (EEG) in RSVP task, the ensemble-model methods have better performance than the single-model ones.Approach.This study proposed a method based on ensemble learning to extract discriminant information of EEG. An extreme gradient boosting framework was utilized to sequentially generate the sub models, including one global spatial-temporal filter and a group of local ones. EEG was reshaped into a three-dimensional form by remapping the electrode dimension into a 2D array to learn the spatial-temporal features from real local space.Main results.A benchmark RSVP EEG dataset was utilized to evaluate the performance of the proposed method, where EEG data of 63 subjects were analyzed. Compared with several state-of-the-art methods, the spatial-temporal patterns of proposed method were more consistent with P300, and the proposed method can provide significantly better classification performance.Significance.The ensemble model in this study was end-to-end optimized, which can avoid error accumulation. The sub models optimized by gradient boosting theory can extract discriminant information complementarily and non-redundantly.

Brain-Computer Interface Based on Steady-State Visual Evoked Potential Using Quick-Response Code Pattern for Wheelchair Control

Brain-computer interfaces (BCIs) are widely utilized in control applications for people with severe physical disabilities. Several researchers have aimed to develop practical brain-controlled wheelchairs. An existing electroencephalogram (EEG)-based BCI based on steady-state visually evoked potential (SSVEP) was developed for device control. This study utilized a quick-response (QR) code visual stimulus pattern for a robust existing system. Four commands were generated using the proposed visual stimulation pattern with four flickering frequencies. Moreover, we employed a relative power spectrum density (PSD) method for the SSVEP feature extraction and compared it with an absolute PSD method. We designed experiments to verify the efficiency of the proposed system. The results revealed that the proposed SSVEP method and algorithm yielded an average classification accuracy of approximately 92% in real-time processing. For the wheelchair simulated via independent-based control, the proposed BCI control required approximately five-fold more time than the keyboard control for real-time control. The proposed SSVEP method using a QR code pattern can be used for BCI-based wheelchair control. However, it suffers from visual fatigue owing to long-time continuous control. We will verify and enhance the proposed system for wheelchair control in people with severe physical disabilities.

Coordinating human-robot collaboration by EEG-based human intention prediction and vigilance control

In human-robot collaboration scenarios with shared workspaces, a highly desired performance boost is offset by high requirements for human safety, limiting speed and torque of the robot drives to levels which cannot harm the human body. Especially for complex tasks with flexible human behavior, it becomes vital to maintain safe working distances and coordinate tasks efficiently. An established approach in this regard is reactive servo in response to the current human pose. However, such an approach does not exploit expectations of the human's behavior and can therefore fail to react to fast human motions in time. To adapt the robot's behavior as soon as possible, predicting human intention early becomes a factor which is vital but hard to achieve. Here, we employ a recently developed type of brain-computer interface (BCI) which can detect the focus of the human's overt attention as a predictor for impending action. In contrast to other types of BCI, direct projection of stimuli onto the workspace facilitates a seamless integration in workflows. Moreover, we demonstrate how the signal-to-noise ratio of the brain response can be used to adjust the velocity of the robot movements to the vigilance or alertness level of the human. Analyzing this adaptive system with respect to performance and safety margins in a physical robot experiment, we found the proposed method could improve both collaboration efficiency and safety distance.

Research on Movement Intentions of Human's Left and Right Legs Based on EEG Signals

Active rehabilitation training method can help stroke patients recover better and faster. However, the lower limb rehabilitation robot based on electroencephalogram (EEG) has low recognition accuracy now. A classification method based on EEG signals of motor imagery is proposed to enable patients to accurately control their left and right legs. Firstly, aiming at the unstable characteristics of EEG signals, an experimental protocl of motor imagery was constructed based on multi-joint motion coupling of left and right legs. The signals with time-frequency analysis and ERD/S analysis have proved the reliability and validity of the collected EEG signals. Then, the EEG signals generated by the protocol were preprocessed and Common Space Pattern (CSP) was used to extract their features. Support Vector Machine (SVM) and Linear Discriminant Analysis (LDA) are adapted and their accuracy of classification results are compared. Finally, on the basis of the proposed classifier with excellent performance, the classifier is used in the active control strategy of the lower limb rehabilitation robot, and the experiment verified that the average accuracy of two volunteers in controlling the lower limb rehabilitation robot reached 95.1%. This research provides a good theoretical basis for the realization and application of brain-computer interface in rehabilitation training.

An asynchronous artifact-enhanced electroencephalogram based control paradigm assisted by slight facial expression

In this study, an asynchronous artifact-enhanced electroencephalogram (EEG)-based control paradigm assisted by slight-facial expressions (sFE-paradigm) was developed. The brain connectivity analysis was conducted to reveal the dynamic directional interactions among brain regions under sFE-paradigm. The component analysis was applied to estimate the dominant components of sFE-EEG and guide the signal processing. Enhanced by the artifact within the detected electroencephalogram (EEG), the sFE-paradigm focused on the mainstream defect as the insufficiency of real-time capability, asynchronous logic, and robustness. The core algorithm contained four steps, including “obvious non-sFE-EEGs exclusion,” “interface ‘ON’ detection,” “sFE-EEGs real-time decoding,” and “validity judgment.” It provided the asynchronous function, decoded eight instructions from the latest 100 ms signal, and greatly reduced the frequent misoperation. In the offline assessment, the sFE-paradigm achieved 96.46% ± 1.07 accuracy for interface “ON” detection and 92.68% ± 1.21 for sFE-EEGs real-time decoding, with the theoretical output timespan less than 200 ms. This sFE-paradigm was applied to two online manipulations for evaluating stability and agility. In “object-moving with a robotic arm,” the averaged intersection-over-union was 60.03 ± 11.53%. In “water-pouring with a prosthetic hand,” the average water volume was 202.5 ± 7.0 ml. During online, the sFE-paradigm performed no significant difference (P = 0.6521 and P = 0.7931) with commercial control methods (i.e., FlexPendant and Joystick), indicating a similar level of controllability and agility. This study demonstrated the capability of sFE-paradigm, enabling a novel solution to the non-invasive EEG-based control in real-world challenges.

A novel brain-controlled wheelchair combined with computer vision and augmented reality

BACKGROUND

Brain-controlled wheelchairs (BCWs) are important applications of brain-computer interfaces (BCIs). Currently, most BCWs are semiautomatic. When users want to reach a target of interest in their immediate environment, this semiautomatic interaction strategy is slow.

METHODS

To this end, we combined computer vision (CV) and augmented reality (AR) with a BCW and proposed the CVAR-BCW: a BCW with a novel automatic interaction strategy. The proposed CVAR-BCW uses a translucent head-mounted display (HMD) as the user interface, uses CV to automatically detect environments, and shows the detected targets through AR technology. Once a user has chosen a target, the CVAR-BCW can automatically navigate to it. For a few scenarios, the semiautomatic strategy might be useful. We integrated a semiautomatic interaction framework into the CVAR-BCW. The user can switch between the automatic and semiautomatic strategies.

RESULTS

We recruited 20 non-disabled subjects for this study and used the accuracy, information transfer rate (ITR), and average time required for the CVAR-BCW to reach each designated target as performance metrics. The experimental results showed that our CVAR-BCW performed well in indoor environments: the average accuracies across all subjects were 83.6% (automatic) and 84.1% (semiautomatic), the average ITRs were 8.2 bits/min (automatic) and 8.3 bits/min (semiautomatic), the average times required to reach a target were 42.4 s (automatic) and 93.4 s (semiautomatic), and the average workloads and degrees of fatigue for the two strategies were both approximately 20.

CONCLUSIONS

Our CVAR-BCW provides a user-centric interaction approach and a good framework for integrating more advanced artificial intelligence technologies, which may be useful in the field of disability assistance.

Humanoid Robot Walking in Maze Controlled by SSVEP-BCI Based on Augmented Reality Stimulus

The application study of robot control based brain-computer interface (BCI) not only helps to promote the practicality of BCI but also helps to promote the advancement of robot technology, which is of great significance. Among the many obstacles, the importability of the stimulator brings much inconvenience to the robot control task. In this study, augmented reality (AR) technology was employed as the visual stimulator of steady-state visual evoked potential (SSVEP)-BCI and the robot walking experiment in the maze was designed to testify the applicability of the AR-BCI system. The online experiment was designed to complete the robot maze walking task and the robot walking commands were sent out by BCI system, in which human intentions were decoded by Filter Bank Canonical Correlation Analysis (FBCCA) algorithm. The results showed that all the 12 subjects could complete the robot walking task in the maze, which verified the feasibility of the AR-SSVEP-NAO system. This study provided an application demonstration for the robot control base on brain–computer interface, and further provided a new method for the future portable BCI system.

Cross-Platform Implementation of an SSVEP-Based BCI for the Control of a 6-DOF Robotic Arm

Robotics has been successfully applied in the design of collaborative robots for assistance to people with motor disabilities. However, man-machine interaction is difficult for those who suffer severe motor disabilities. The aim of this study was to test the feasibility of a low-cost robotic arm control system with an EEG-based brain-computer interface (BCI). The BCI system relays on the Steady State Visually Evoked Potentials (SSVEP) paradigm. A cross-platform application was obtained in C++. This C++ platform, together with the open-source software Openvibe was used to control a Stäubli robot arm model TX60. Communication between Openvibe and the robot was carried out through the Virtual Reality Peripheral Network (VRPN) protocol. EEG signals were acquired with the 8-channel Enobio amplifier from Neuroelectrics. For the processing of the EEG signals, Common Spatial Pattern (CSP) filters and a Linear Discriminant Analysis classifier (LDA) were used. Five healthy subjects tried the BCI. This work allowed the communication and integration of a well-known BCI development platform such as Openvibe with the specific control software of a robot arm such as Stäubli TX60 using the VRPN protocol. It can be concluded from this study that it is possible to control the robotic arm with an SSVEP-based BCI with a reduced number of dry electrodes to facilitate the use of the system.

Manual 3D Control of an Assistive Robotic Manipulator Using Alpha Rhythms and an Auditory Menu: A Proof-of-Concept

Brain–Computer Interfaces (BCIs) have been regarded as potential tools for individuals with severe motor disabilities, such as those with amyotrophic lateral sclerosis, that render interfaces that rely on movement unusable. This study aims to develop a dependent BCI system for manual end-point control of a robotic arm. A proof-of-concept system was devised using parieto-occipital alpha wave modulation and a cyclic menu with auditory cues. Users choose a movement to be executed and asynchronously stop said action when necessary. Tolerance intervals allowed users to cancel or confirm actions. Eight able-bodied subjects used the system to perform a pick-and-place task. To investigate the potential learning effects, the experiment was conducted twice over the course of two consecutive days. Subjects obtained satisfactory completion rates (84.0 ± 15.0% and 74.4 ± 34.5% for the first and second day, respectively) and high path efficiency (88.9 ± 11.7% and 92.2 ± 9.6%). Subjects took on average 439.7 ± 203.3 s to complete each task, but the robot was only in motion 10% of the time. There was no significant difference in performance between both days. The developed control scheme provided users with intuitive control, but a considerable amount of time is spent waiting for the right target (auditory cue). Implementing other brain signals may increase its speed.

Riemannian geometry-based transfer learning for reducing training time in c-VEP BCIs

One of the main problems that a brain-computer interface (BCI) face is that a training stage is required for acquiring training data to calibrate its classification model just before every use. Transfer learning is a promising method for addressing the problem. In this paper, we propose a Riemannian geometry-based transfer learning algorithm for code modulated visual evoked potential (c-VEP)-based BCIs, which can effectively reduce the calibration time without sacrificing the classification accuracy. The algorithm includes the main procedures of log-Euclidean data alignment (LEDA), super-trial construction, covariance matrix estimation, training accuracy-based subject selection (TSS) and minimum distance to mean classification. Among them, the LEDA reduces the difference in data distribution between subjects, whereas the TSS promotes the similarity between a target subject and the source subjects. The resulting performance of transfer learning is improved significantly. Sixteen subjects participated in a c-VEP BCI experiment and the recorded data were used in offline analysis. Leave-one subject-out (LOSO) cross-validation was used to evaluate the proposed algorithm on the data set. The results showed that the algorithm achieved much higher classification accuracy than the subject-specific (baseline) algorithm with the same number of training trials. Equivalently, the algorithm reduces the training time of the BCI at the same performance level and thus facilitates its application in real world.

A solution to supervised motor imagery task in the BCI Controlled Robot Contest in World Robot Contest

Background:

One of the most prestigious competitions in the world is the World Robot Conference. This paper presents the winning solution to the supervised motor imagery (MI) task in the BCI Controlled Robot Contest in World Robot Contest 2021.

Methods:

Data augmentation, preprocessing, feature extraction, and model training are the main components of the solution. The model is based on EEGNet, a popular convolutional neural networks model for classifying electroencephalography data.

Results:

Despite the model’s lack of stability, this solution was the most successful in the task. The channels closest to the vertex were the most helpful in feature extraction.

Conclusion:

This solution is suitable for supervised MI tasks not only in this competition but also in future scenarios.

eldBETA: A Large Eldercare-oriented Benchmark Database of SSVEP-BCI for the Aging Population

Global population aging poses an unprecedented challenge and calls for a rising effort in eldercare and healthcare. Steady-state visual evoked potential based brain-computer interface (SSVEP-BCI) boasts its high transfer rate and shows great promise in real-world applications to support aging. Public database is critically important for designing the SSVEP-BCI systems. However, the SSVEP-BCI database tailored for the elder is scarce in existing studies. Therefore, in this study, we present a large eldercare-oriented BEnchmark database of SSVEP-BCI for The Aging population (eldBETA). The eldBETA database consisted of the 64-channel electroencephalogram (EEG) from 100 elder participants, each of whom performed seven blocks of 9-target SSVEP-BCI task. The quality and characteristics of the eldBETA database were validated by a series of analyses followed by a classification analysis of thirteen frequency recognition methods. We expect that the eldBETA database would provide a substrate for the design and optimization of the BCI systems intended for the elders. The eldBETA database is open-access for research and can be downloaded from the website 10.6084/m9.figshare.18032669.

Measurement(s)	Steady-state visual evoked potential (SSVEP)
Technology Type(s)	Electroencephalography (EEG)
Factor Type(s)	Elder population
Sample Characteristic - Organism	Homo sapiens
Sample Characteristic - Environment	Electromagnetic shielding room

An MVMD-CCA Recognition Algorithm in SSVEP-Based BCI and Its Application in Robot Control

This article proposes a novel recognition algorithm for the steady-state visual evoked potentials (SSVEP)-based brain-computer interface (BCI) system. By combining the advantages of multivariate variational mode decomposition (MVMD) and canonical correlation analysis (CCA), an MVMD-CCA algorithm is investigated to improve the detection ability of SSVEP electroencephalogram (EEG) signals. In comparison with the classical filter bank canonical correlation analysis (FBCCA), the nonlinear and non-stationary EEG signals are decomposed into a fixed number of sub-bands by MVMD, which can enhance the effect of SSVEP-related sub-bands. The experimental results show that MVMD-CCA can effectively reduce the influence of noise and EEG artifacts and improve the performance of SSVEP-based BCI. The offline experiments show that the average accuracies of MVMD-CCA in the training dataset and testing dataset are improved by 3.08% and 1.67%, respectively. In the SSVEP-based online robotic manipulator grasping experiment, the recognition accuracies of the four subjects are 92.5%, 93.33%, 90.83%, and 91.67%, respectively.

A Noninvasive BCI System for 2D Cursor Control Using a Spectral-Temporal Long Short-Term Memory Network

Two-dimensional cursor control is an important and challenging problem in the field of electroencephalography (EEG)-based brain computer interfaces (BCIs) applications. However, most BCIs based on categorical outputs are incapable of generating accurate and smooth control trajectories. In this article, a novel EEG decoding framework based on a spectral-temporal long short-term memory (stLSTM) network is proposed to generate control signals in the horizontal and vertical directions for accurate cursor control. Precisely, the spectral information is used to decode the subject's motor imagery intention, and the error-related P300 information is used to detect a deviation in the movement trajectory. The concatenated spectral and temporal features are fed into the stLSTM network and mapped to the velocities in vertical and horizontal directions of the 2D cursor under the velocity-constrained (VC) strategy, which enables the decoding network to fit the velocity in the imaginary direction and simultaneously suppress the velocity in the non-imaginary direction. This proposed framework was validated on a public real BCI control dataset. Results show that compared with the state-of-the-art method, the RMSE of the proposed method in the non-imaginary directions on the testing sets of 2D control tasks is reduced by an average of 63.45%. Besides, the visualization of the actual trajectories distribution of the cursor also demonstrates that the decoupling of velocity is capable of yielding accurate cursor control in complex path tracking tasks and significantly improves the control accuracy.

A Human-Machine Interface Based on an EOG and a Gyroscope for Humanoid Robot Control and Its Application to Home Services

The human-machine interface (HMI) has been studied for robot teleoperation with the aim of empowering people who experience motor disabilities to increase their interaction with the physical environment. The challenge of an HMI for robot control is to rapidly, accurately, and sufficiently produce control commands. In this paper, an asynchronous HMI based on an electrooculogram (EOG) and a gyroscope is proposed using two self-paced and endogenous features, double blink and head rotation. By designing the multilevel graphical user interface (GUI), the user can rotate his head to move the cursor of the GUI and create a double blink to trigger the button in the interface. The proposed HMI is able to supply sufficient commands at the same time with high accuracy (ACC) and low response time (RT). In the trigger task of sixteen healthy subjects, the target was clicked from 20 options with ACC of 99.2% and RT 2.34 s. Furthermore, a continuous strategy that uses motion start and motion stop commands to create a certain robot motion is proposed to control a humanoid robot based on the HMI. It avoids the situation that combines some commands to achieve one motion or converts the certain motion to a command directly. In the home service experiment, all subjects operated a humanoid robot changing the state of a switch, grasping a key, and putting it into a box. The time ratio between HMI control and manual control was 1.22, and the number of commands ratio was 1.18. The results demonstrated that the continuous strategy and proposed HMI can improve performance in humanoid robot control.

Control of a Robotic Arm With an Optimized Common Template-Based CCA Method for SSVEP-Based BCI

Recently, the robotic arm control system based on a brain-computer interface (BCI) has been employed to help the disabilities to improve their interaction abilities without body movement. However, it's the main challenge to implement the desired task by a robotic arm in a three-dimensional (3D) space because of the instability of electroencephalogram (EEG) signals and the interference by the spontaneous EEG activities. Moreover, the free motion control of a manipulator in 3D space is a complicated operation that requires more output commands and higher accuracy for brain activity recognition. Based on the above, a steady-state visual evoked potential (SSVEP)-based synchronous BCI system with six stimulus targets was designed to realize the motion control function of the seven degrees of freedom (7-DOF) robotic arm. Meanwhile, a novel template-based method, which builds the optimized common templates (OCTs) from various subjects and learns spatial filters from the common templates and the multichannel EEG signal, was applied to enhance the SSVEP recognition accuracy, called OCT-based canonical correlation analysis (OCT-CCA). The comparison results of offline experimental based on a public benchmark dataset indicated that the proposed OCT-CCA method achieved significant improvement of detection accuracy in contrast to CCA and individual template-based CCA (IT-CCA), especially using a short data length. In the end, online experiments with five healthy subjects were implemented for achieving the manipulator real-time control system. The results showed that all five subjects can accomplish the tasks of controlling the manipulator to reach the designated position in the 3D space independently.

Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking

The controlling of robotic arms based on brain–computer interface (BCI) can revolutionize the quality of life and living conditions for individuals with physical disabilities. Invasive electroencephalography (EEG)-based BCI has been able to control multiple degrees of freedom (DOFs) robotic arms in three dimensions. However, it is still hard to control a multi-DOF robotic arm to reach and grasp the desired target accurately in complex three-dimensional (3D) space by a noninvasive system mainly due to the limitation of EEG decoding performance. In this study, we propose a noninvasive EEG-based BCI for a robotic arm control system that enables users to complete multitarget reach and grasp tasks and avoid obstacles by hybrid control. The results obtained from seven subjects demonstrated that motor imagery (MI) training could modulate brain rhythms, and six of them completed the online tasks using the hybrid-control-based robotic arm system. The proposed system shows effective performance due to the combination of MI-based EEG, computer vision, gaze detection, and partially autonomous guidance, which drastically improve the accuracy of online tasks and reduce the brain burden caused by long-term mental activities.

Customizing skills for assistive robotic manipulators, an inverse reinforcement learning approach with error-related potentials

Robotic assistance via motorized robotic arm manipulators can be of valuable assistance to individuals with upper-limb motor disabilities. Brain-computer interfaces (BCI) offer an intuitive means to control such assistive robotic manipulators. However, BCI performance may vary due to the non-stationary nature of the electroencephalogram (EEG) signals. It, hence, cannot be used safely for controlling tasks where errors may be detrimental to the user. Avoiding obstacles is one such task. As there exist many techniques to avoid obstacles in robotics, we propose to give the control to the robot to avoid obstacles and to leave to the user the choice of the robot behavior to do so a matter of personal preference as some users may be more daring while others more careful. We enable the users to train the robot controller to adapt its way to approach obstacles relying on BCI that detects error-related potentials (ErrP), indicative of the user’s error expectation of the robot’s current strategy to meet their preferences. Gaussian process-based inverse reinforcement learning, in combination with the ErrP-BCI, infers the user’s preference and updates the obstacle avoidance controller so as to generate personalized robot trajectories. We validate the approach in experiments with thirteen able-bodied subjects using a robotic arm that picks up, places and avoids real-life objects. Results show that the algorithm can learn user’s preference and adapt the robot behavior rapidly using less than five demonstrations not necessarily optimal.

Teaching an assistive robotic manipulator to move objects in a cluttered table requires demonstrations from expert operators, but what if the experts are individuals with motor disabilities? Batzianoulis et al. propose a learning approach which combines robot autonomy and a brain-computer interfacing that decodes whether the generated trajectories meet the user’s criteria, and show how their system enables the robot to learn individual user’s preferred behaviors using less than five demonstrations that are not necessarily optimal.

Artificial Intelligence Algorithms in Visual Evoked Potential-Based Brain-Computer Interfaces for Motor Rehabilitation Applications: Systematic Review and Future Directions

Brain-Computer Interface (BCI) is a technology that uses electroencephalographic (EEG) signals to control external devices, such as Functional Electrical Stimulation (FES). Visual BCI paradigms based on P300 and Steady State Visually Evoked potentials (SSVEP) have shown high potential for clinical purposes. Numerous studies have been published on P300- and SSVEP-based non-invasive BCIs, but many of them present two shortcomings: (1) they are not aimed for motor rehabilitation applications, and (2) they do not report in detail the artificial intelligence (AI) methods used for classification, or their performance metrics. To address this gap, in this paper the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) methodology was applied to prepare a systematic literature review (SLR). Papers older than 10 years, repeated or not related to a motor rehabilitation application, were excluded. Of all the studies, 51.02% referred to theoretical analysis of classification algorithms. Of the remaining, 28.48% were for spelling, 12.73% for diverse applications (control of wheelchair or home appliances), and only 7.77% were focused on motor rehabilitation. After the inclusion and exclusion criteria were applied and quality screening was performed, 34 articles were selected. Of them, 26.47% used the P300 and 55.8% the SSVEP signal. Five applications categories were established: Rehabilitation Systems (17.64%), Virtual Reality environments (23.52%), FES (17.64%), Orthosis (29.41%), and Prosthesis (11.76%). Of all the works, only four performed tests with patients. The most reported machine learning (ML) algorithms used for classification were linear discriminant analysis (LDA) (48.64%) and support vector machine (16.21%), while only one study used a deep learning algorithm: a Convolutional Neural Network (CNN). The reported accuracy ranged from 38.02 to 100%, and the Information Transfer Rate from 1.55 to 49.25 bits per minute. While LDA is still the most used AI algorithm, CNN has shown promising results, but due to their high technical implementation requirements, many researchers do not justify its implementation as worthwile. To achieve quick and accurate online BCIs for motor rehabilitation applications, future works on SSVEP-, P300-based and hybrid BCIs should focus on optimizing the visual stimulation module and the training stage of ML and DL algorithms.

A CNN-based multi-target fast classification method for AR-SSVEP

Because an augmented-reality-based brain-computer interface (AR-BCI) is easily disturbed by external factors, the traditional electroencephalograph (EEG) classification algorithms fail to meet the real-time processing requirements with a large number of stimulus targets or in a real environment. We propose a multi-target fast classification method for augmented-reality-based steady-state visual evoked potential (AR-SSVEP), using a convolutional neural network (CNN). To explore the availability and accuracy of high-efficiency multi-target classification methods in AR-SSVEP with a short stimulation duration, a similar stimulus layout was used for a computer screen (PC) and an optical see-through head-mounted display (OST-HMD) device (HoloLens). The experiment included nine flicker stimuli of different frequencies, and a multi-target fast classification method based on a CNN was constructed to complete nine classification tasks, for which the average accuracy of AR-BCI in our CNN model at 0.5- and 1-s stimulus duration was 67.93% and 80.83%, respectively. These results verified the efficacy of the proposed model for processing multi-target classification in AR-BCI.

Adaptive asynchronous control system of robotic arm based on augmented reality-assisted brain-computer interface

Objective. Brain-controlled robotic arms have shown broad application prospects with the development of robotics, science and information decoding. However, disadvantages, such as poor flexibility restrict its wide application.Approach. In order to alleviate these drawbacks, this study proposed a robotic arm asynchronous control system based on steady-state visual evoked potential (SSVEP) in an augmented reality (AR) environment. In the AR environment, the participants were able to concurrently see the robot arm and visual stimulation interface through the AR device. Therefore, there was no need to switch attention frequently between the visual stimulation interface and the robotic arm. This study proposed a multi-template algorithm based on canonical correlation analysis and task-related component analysis to identify 12 targets. An optimization strategy based on dynamic window was adopted to adjust the duration of visual stimulation adaptively.Main results. Experimental results of this study found that the high-frequency SSVEP-based brain-computer interface (BCI) realized the switch of the system state, which controlled the robotic arm asynchronously. The average accuracy of the offline experiment was 94.97%, whereas the average information translate rate was 67.37 ± 14.27 bits·min^-1. The online results from ten healthy subjects showed that the average selection time of a single online command was 2.04 s, which effectively reduced the visual fatigue of the subjects. Each subject could quickly complete the puzzle task.Significance. The experimental results demonstrated the feasibility and potential of this human-computer interaction strategy and provided new ideas for BCI-controlled robots.

Emergence of flexible technology in developing advanced systems for post-stroke rehabilitation: a comprehensive review

OBJECTIVE

Stroke is one of the most common neural disorders, which causes physical disabilities and motor impairments among its survivors. Several technologies have been developed for providing stroke rehabilitation and to assist the survivors in performing their daily life activities. Currently, the use of flexible technology (FT) for stroke rehabilitation systems is on a rise that allows the development of more compact and lightweight wearable systems, which stroke survivors can easily use for long-term activities.

APPROACH

For stroke applications, FT mainly includes the "flexible/stretchable electronics", "e-textile (electronic textile)" and "soft robotics". Thus, a thorough literature review has been performed to report the practical implementation of FT for post-stroke application.

MAIN RESULTS

In this review, the highlights of the advancement of FT in stroke rehabilitation systems are dealt with. Such systems mainly involve the "biosignal acquisition unit", "rehabilitation devices" and "assistive systems". In terms of biosignals acquisition, electroencephalography (EEG) and electromyography (EMG) are comprehensively described. For rehabilitation/assistive systems, the application of functional electrical stimulation (FES) and robotics units (exoskeleton, orthosis, etc.) have been explained.

SIGNIFICANCE

This is the first review article that compiles the different studies regarding flexible technology based post-stroke systems. Furthermore, the technological advantages, limitations, and possible future implications are also discussed to help improve and advance the flexible systems for the betterment of the stroke community.

Motor Intention Decoding from the Upper Limb by Graph Convolutional Network Based on Functional Connectivity

Decoding brain intention from noninvasively measured neural signals has recently been a hot topic in brain-computer interface (BCI). The motor commands about the movements of fine parts can increase the degrees of freedom under control and be applied to external equipment without stimulus. In the decoding process, the classifier is one of the key factors, and the graph information of the EEG was ignored by most researchers. In this paper, a graph convolutional network (GCN) based on functional connectivity was proposed to decode the motor intention of four fine parts movements (shoulder, elbow, wrist, hand). First, event-related desynchronization was analyzed to reveal the differences between the four classes. Second, functional connectivity was constructed by using synchronization likelihood (SL), phase-locking value (PLV), H index (H), mutual information (MI), and weighted phase-lag index (WPLI) to acquire the electrode pairs with a difference. Subsequently, a GCN and convolutional neural networks (CNN) were performed based on functional topological structures and time points, respectively. The results demonstrated that the proposed method achieved a decoding accuracy of up to 92.81% in the four-class task. Besides, the combination of GCN and functional connectivity can promote the development of BCI.

Brain-Computer Interface: Advancement and Challenges

Brain-Computer Interface (BCI) is an advanced and multidisciplinary active research domain based on neuroscience, signal processing, biomedical sensors, hardware, etc. Since the last decades, several groundbreaking research has been conducted in this domain. Still, no comprehensive review that covers the BCI domain completely has been conducted yet. Hence, a comprehensive overview of the BCI domain is presented in this study. This study covers several applications of BCI and upholds the significance of this domain. Then, each element of BCI systems, including techniques, datasets, feature extraction methods, evaluation measurement matrices, existing BCI algorithms, and classifiers, are explained concisely. In addition, a brief overview of the technologies or hardware, mostly sensors used in BCI, is appended. Finally, the paper investigates several unsolved challenges of the BCI and explains them with possible solutions.

A brain-actuated robotic arm system using non-invasive hybrid brain-computer interface and shared control strategy

Objective.The electroencephalography (EEG)-based brain-computer interfaces (BCIs) have been used in the control of robotic arms. The performance of non-invasive BCIs may not be satisfactory due to the poor quality of EEG signals, so the shared control strategies were tried as an alternative solution. However, most of the existing shared control methods set the arbitration rules manually, which highly depended on the specific tasks and developer's experience. In this study, we proposed a novel shared control model that automatically optimized the control commands in a dynamical way based on the context in real-time control. Besides, we employed the hybrid BCI to better allocate commands with multiple functions. The system allowed non-invasive BCI users to manipulate a robotic arm moving in a three-dimensional (3D) space and complete a pick-place task of multiple objects.Approach.Taking the scene information obtained by computer vision as a knowledge base, a machine agent was designed to infer the user's intention and generate automatic commands. Based on the inference confidence and user's characteristic, the proposed shared control model fused the machine autonomy and human intention dynamically for robotic arm motion optimization during the online control. In addition, we introduced a hybrid BCI scheme that applied steady-state visual evoked potentials and motor imagery to the divided primary and secondary BCI interfaces to better allocate the BCI resources (e.g. decoding computing power, screen occupation) and realize the multi-dimensional control of the robotic arm.Main results.Eleven subjects participated in the online experiments of picking and placing five objects that scattered at different positions in a 3D workspace. The results showed that most of the subjects could control the robotic arm to complete accurate and robust picking task with an average success rate of approximately 85% under the shared control strategy, while the average success rate of placing task controlled by pure BCI was 50% approximately.Significance.In this paper, we proposed a novel shared controller for motion automatic optimization, together with a hybrid BCI control scheme that allocated paradigms according to the importance of commands to realize multi-dimensional and effective control of a robotic arm. Our study indicated that the shared control strategy with hybrid BCI could greatly improve the performance of the brain-actuated robotic arm system.

Reconstruction of hand, elbow and shoulder actual and imagined trajectories in 3D space using EEG current source dipoles

OBJECTIVE

Growing evidence suggests that EEG electrode (sensor) potential time series (PTS) of slow cortical potentials (SCPs) hold motor neural correlates that can be used for motion trajectory prediction (MTP), commonly by multiple linear regression (mLR). It is not yet known whether arm-joint trajectories can be reliably decoded from current sources, computed from sensor data, from which brain areas they can be decoded and using which neural features.

APPROACH

In this study, the PTS of 44 sensors were fed into sLORETA source localization software to compute current source activity in 30 regions of interest (ROIs) found in a recent meta-analysis to be engaged in action execution, motor imagery and motor preparation. The current sources PTS and band-power time series (BTS) in several frequency bands and time lags were used to predict actual and imagined trajectories in 3D space of the three velocity components of the hand, elbow and shoulder of nine subjects using an mLR model.

MAIN RESULTS

For all arm joints and movement types, current source SCPs PTS contributed most to trajectory reconstruction with time lags 150ms, 116ms and 84ms providing the highest contribution, and current source BTS in any of the tested frequency bands was not informative. Person's correlation coefficient (r) averaged across movement types, arm joints and velocity components using source data was slightly lower than using sensor data (r=0.25 and r=0.28, respectively). For each ROI, the three current source dipoles had different contribution to the reconstruction of each of the three velocity components.

SIGNIFICANCE

Overall, our results demonstrate the feasibility of predicting of actual and imagined 3D trajectories of all arm joints from current sources, computed from scalp EEG. These findings may be used by developers of a future BCI as a validated set of contributing ROIs.

Motor Training Using Mental Workload (MWL) With an Assistive Soft Exoskeleton System: A Functional Near-Infrared Spectroscopy (fNIRS) Study for Brain-Machine Interface (BMI)

Mental workload is a neuroergonomic human factor, which is widely used in planning a system's safety and areas like brain-machine interface (BMI), neurofeedback, and assistive technologies. Robotic prosthetics methodologies are employed for assisting hemiplegic patients in performing routine activities. Assistive technologies' design and operation are required to have an easy interface with the brain with fewer protocols, in an attempt to optimize mobility and autonomy. The possible answer to these design questions may lie in neuroergonomics coupled with BMI systems. In this study, two human factors are addressed: designing a lightweight wearable robotic exoskeleton hand that is used to assist the potential stroke patients with an integrated portable brain interface using mental workload (MWL) signals acquired with portable functional near-infrared spectroscopy (fNIRS) system. The system may generate command signals for operating a wearable robotic exoskeleton hand using two-state MWL signals. The fNIRS system is used to record optical signals in the form of change in concentration of oxy and deoxygenated hemoglobin (HbO and HbR) from the pre-frontal cortex (PFC) region of the brain. Fifteen participants participated in this study and were given hand-grasping tasks. Two-state MWL signals acquired from the PFC region of the participant's brain are segregated using machine learning classifier-support vector machines (SVM) to utilize in operating a robotic exoskeleton hand. The maximum classification accuracy is 91.31%, using a combination of mean-slope features with an average information transfer rate (ITR) of 1.43. These results show the feasibility of a two-state MWL (fNIRS-based) robotic exoskeleton hand (BMI system) for hemiplegic patients assisting in the physical grasping tasks.

Channel Selection for Optimal EEG Measurement in Motor Imagery-Based Brain-Computer Interfaces

A method for selecting electroencephalographic (EEG) signals in motor imagery-based brain-computer interfaces (MI-BCI) is proposed for enhancing the online interoperability and portability of BCI systems, as well as user comfort. The attempt is also to reduce variability and noise of MI-BCI, which could be affected by a large number of EEG channels. The relation between selected channels and MI-BCI performance is therefore analyzed. The proposed method is able to select acquisition channels common to all subjects, while achieving a performance compatible with the use of all the channels. Results are reported with reference to a standard benchmark dataset, the BCI competition IV dataset 2a. They prove that a performance compatible with the best state-of-the-art approaches can be achieved, while adopting a significantly smaller number of channels, both in two and in four tasks classification. In particular, classification accuracy is about 77-83% in binary classification with down to 6 EEG channels, and above 60% for the four-classes case when 10 channels are employed. This gives a contribution in optimizing the EEG measurement while developing non-invasive and wearable MI-based brain-computer interfaces.

A Training Data-Driven Canonical Correlation Analysis Algorithm for Designing Spatial Filters to Enhance Performance of SSVEP-Based BCIs

Canonical correlation analysis (CCA) is an effective spatial filtering algorithm widely used in steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs). In existing CCA methods, training data are used for constructing templates of stimulus targets and the spatial filters are created between the template signals and a single-trial testing signal. The fact that spatial filters rely on testing data, however, results in low classification performance of CCA compared to other state-of-the-art algorithms such as task-related component analysis (TRCA). In this study, we proposed a novel CCA method in which spatial filters are estimated using training data only. This is achieved by using observed EEG training data and their SSVEP components as the two inputs of CCA and the objective function is optimized by averaging multiple training trials. In this case, we proved in theory that the two spatial filters estimated by the CCA are equivalent, and that the CCA and TRCA are also equivalent under certain hypotheses. A benchmark SSVEP data set from 35 subjects was used to compare the performance of the two algorithms according to different lengths of data, numbers of channels and numbers of training trials. In addition, the CCA was also compared with power spectral density analysis (PSDA). The experimental results suggest that the CCA is equivalent to TRCA if the signal-to-noise ratio of training data is high enough; otherwise, the CCA outperforms TRCA in terms of classification accuracy. The CCA is much faster than PSDA in detecting time of targets. The robustness of the training data-driven CCA to noise gives it greater potential in practical applications.

A Hybrid BCI Based on SSVEP and EOG for Robotic Arm Control

Brain-computer interface (BCI) for robotic arm control has been studied to improve the life quality of people with severe motor disabilities. There are still challenges for robotic arm control in accomplishing a complex task with a series of actions. An efficient switch and a timely cancel command are helpful in the application of robotic arm. Based on the above, we proposed an asynchronous hybrid BCI in this study. The basic control of a robotic arm with six degrees of freedom was a steady-state visual evoked potential (SSVEP) based BCI with fifteen target classes. We designed an EOG-based switch which used a triple blink to either activate or deactivate the flash of SSVEP-based BCI. Stopping flash in the idle state can help to reduce visual fatigue and false activation rate (FAR). Additionally, users were allowed to cancel the current command simply by a wink in the feedback phase to avoid executing the incorrect command. Fifteen subjects participated and completed the experiments. The cue-based experiment obtained an average accuracy of 92.09%, and the information transfer rates (ITR) resulted in 35.98 bits/min. The mean FAR of the switch was 0.01/min. Furthermore, all subjects succeeded in asynchronously operating the robotic arm to grasp, lift, and move a target object from the initial position to a specific location. The results indicated the feasibility of the combination of EOG and SSVEP signals and the flexibility of EOG signal in BCI to complete a complicated task of robotic arm control.

Convolutional Correlation Analysis for Enhancing the Performance of SSVEP-Based Brain-Computer Interface

Currently, most of the high-performance models for frequency recognition of steady-state visual evoked potentials (SSVEPs) are linear. However, SSVEPs collected from different channels can have non-linear relationship among each other. Linearly combining electroencephalogram (EEG) from multiple channels is not the most accurate solution in SSVEPs classification. To further improve the performance of SSVEP-based brain-computer interface (BCI), we propose a convolutional neural network-based non-linear model, i.e. convolutional correlation analysis (Conv-CA). Different from pure deep learning models, Conv-CA use convolutional neural networks (CNNs) at the top of a self-defined correlation layer. The CNNs function on how to transform multiple channel EEGs into a single EEG signal. The correlation layer calculates the correlation coefficients between the transformed single EEG signal and reference signals. The CNNs provide non-linear operations to combine EEGs in different channels and different time. And the correlation layer constrains the fitting space of the deep learning model. A comparison study between the proposed Conv-CA method and the task-related component analysis (TRCA) based methods is conducted. Both methods are validated on a 40-class SSVEP benchmark dataset recorded from 35 subjects. The study verifies that the Conv-CA method significantly outperforms the TRCA-based methods. Moreover, Conv-CA has good explainability since its inputs of the correlation layer can be analyzed for visualizing what the model learnt from the data. Conv-CA is a non-linear extension of spatial filters. Its CNN structures can be further explored and tuned for reaching a better performance. The structure of combining neural networks and unsupervised features has the potential to be applied to the classification of other signals.

Combination of Augmented Reality Based Brain- Computer Interface and Computer Vision for High-Level Control of a Robotic Arm

Recent advances in robotics, neuroscience, and signal processing make it possible to operate a robot through electroencephalography (EEG)-based brain-computer interface (BCI). Although some successful attempts have been made in recent years, the practicality of the entire system still has much room for improvement. The present study designed and realized a robotic arm control system by combing augmented reality (AR), computer vision, and steady-state visual evoked potential (SSVEP)-BCI. AR environment was implemented by a Microsoft HoloLens. Flickering stimuli for eliciting SSVEPs were presented on the HoloLens, which allowed users to see both the robotic arm and the user interface of the BCI. Thus users did not need to switch attention between the visual stimulator and the robotic arm. A four-command SSVEP-BCI was built for users to choose the specific object to be operated by the robotic arm. Once an object was selected, the computer vision would provide the location and color of the object in the workspace. Subsequently, the object was autonomously picked up and placed by the robotic arm. According to the online results obtained from twelve participants, the mean classification accuracy of the proposed system was 93.96 ± 5.05%. Moreover, all subjects could utilize the proposed system to successfully pick and place objects in a specific order. These results demonstrated the potential of combining AR-BCI and computer vision to control robotic arms, which is expected to further promote the practicality of BCI-controlled robots.