Spatial-temporal aspects of continuous EEG-based neurorobotic control

Continuous shared control of a mobile robot with brain-computer interface and autonomous navigation for daily assistance

Although the electroencephalography (EEG) based brain-computer interface (BCI) has been successfully developed for rehabilitation and assistance, it is still challenging to achieve continuous control of a brain-actuated mobile robot system. In this study, we propose a continuous shared control strategy combining continuous BCI and autonomous navigation for a mobile robot system. The weight of shared control is designed to dynamically adjust the fusion of continuous BCI control and autonomous navigation. During this process, the system uses the visual-based simultaneous localization and mapping (SLAM) method to construct environmental maps. After obtaining the global optimal path, the system utilizes the brain-based shared control dynamic window approach (BSC-DWA) to evaluate safe and reachable trajectories while considering shared control velocity. Eight subjects participated in two-stage training, and six of these eight subjects participated in online shared control experiments. The training results demonstrated that naïve subjects could achieve continuous control performance with an average percent valid correct rate of approximately 97 % and an average total correct rate of over 80 %. The results of online shared control experiments showed that all of the subjects could complete navigation tasks in an unknown corridor with continuous shared control. Therefore, our experiments verified the feasibility and effectiveness of the proposed system combining continuous BCI, shared control, autonomous navigation, and visual SLAM. The proposed continuous shared control framework shows great promise in BCI-driven tasks, especially navigation tasks for brain-driven assistive mobile robots and wheelchairs in daily applications.

Closed-loop motor imagery EEG simulation for brain-computer interfaces

In a brain-computer interface (BCI) system, the testing of decoding algorithms, tasks, and their parameters is critical for optimizing performance. However, conducting human experiments can be costly and time-consuming, especially when investigating broad sets of parameters. Attempts to utilize previously collected data in offline analysis lack a co-adaptive feedback loop between the system and the user present online, limiting the applicability of the conclusions obtained to real-world uses of BCI. As such, a number of studies have attempted to address this cost-wise middle ground between offline and live experimentation with real-time neural activity simulators. We present one such system which generates motor imagery electroencephalography (EEG) via forward modeling and novel motor intention encoding models for conducting sensorimotor rhythm (SMR)-based continuous cursor control experiments in a closed-loop setting. We use the proposed simulator with 10 healthy human subjects to test the effect of three decoder and task parameters across 10 different values. Our simulated approach produces similar statistical conclusions to those produced during parallel, paired, online experimentation, but in 55% of the time. Notably, both online and simulated experimentation expressed a positive effect of cursor velocity limit on performance regardless of subject average performance, supporting the idea of relaxing constraints on cursor gain in online continuous cursor control. We demonstrate the merits of our closed-loop motor imagery EEG simulation, and provide an open-source framework to the community for closed-loop SMR-based BCI studies in the future. All code including the simulator have been made available on GitHub.

A Noninvasive BCI System for 2D Cursor Control Using a Spectral-Temporal Long Short-Term Memory Network

Two-dimensional cursor control is an important and challenging problem in the field of electroencephalography (EEG)-based brain computer interfaces (BCIs) applications. However, most BCIs based on categorical outputs are incapable of generating accurate and smooth control trajectories. In this article, a novel EEG decoding framework based on a spectral-temporal long short-term memory (stLSTM) network is proposed to generate control signals in the horizontal and vertical directions for accurate cursor control. Precisely, the spectral information is used to decode the subject's motor imagery intention, and the error-related P300 information is used to detect a deviation in the movement trajectory. The concatenated spectral and temporal features are fed into the stLSTM network and mapped to the velocities in vertical and horizontal directions of the 2D cursor under the velocity-constrained (VC) strategy, which enables the decoding network to fit the velocity in the imaginary direction and simultaneously suppress the velocity in the non-imaginary direction. This proposed framework was validated on a public real BCI control dataset. Results show that compared with the state-of-the-art method, the RMSE of the proposed method in the non-imaginary directions on the testing sets of 2D control tasks is reduced by an average of 63.45%. Besides, the visualization of the actual trajectories distribution of the cursor also demonstrates that the decoupling of velocity is capable of yielding accurate cursor control in complex path tracking tasks and significantly improves the control accuracy.

Functional connectivity ensemble method to enhance BCI performance (FUCONE)

OBJECTIVE

Relying on the idea that functional connectivity provides important insights on the underlying dynamic of neuronal interactions, we propose a novel framework that combines functional connectivity estimators and covariance-based pipelines to improve the classification of mental states, such as motor imagery.

METHODS

A Riemannian classifier is trained for each estimator and an ensemble classifier combines the decisions in each feature space. A thorough assessment of the functional connectivity estimators is provided and the best performing pipeline among those tested, called FUCONE, is evaluated on different conditions and datasets.

RESULTS

Using a meta-analysis to aggregate results across datasets, FUCONE performed significantly better than all state-of-the-art methods.

CONCLUSION

The performance gain is mostly imputable to the improved diversity of the feature spaces, increasing the robustness of the ensemble classifier with respect to the inter- and intra-subject variability.

SIGNIFICANCE

Our results offer new insights into the need to consider functional connectivity-based methods to improve the BCI performance.

Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking

The controlling of robotic arms based on brain–computer interface (BCI) can revolutionize the quality of life and living conditions for individuals with physical disabilities. Invasive electroencephalography (EEG)-based BCI has been able to control multiple degrees of freedom (DOFs) robotic arms in three dimensions. However, it is still hard to control a multi-DOF robotic arm to reach and grasp the desired target accurately in complex three-dimensional (3D) space by a noninvasive system mainly due to the limitation of EEG decoding performance. In this study, we propose a noninvasive EEG-based BCI for a robotic arm control system that enables users to complete multitarget reach and grasp tasks and avoid obstacles by hybrid control. The results obtained from seven subjects demonstrated that motor imagery (MI) training could modulate brain rhythms, and six of them completed the online tasks using the hybrid-control-based robotic arm system. The proposed system shows effective performance due to the combination of MI-based EEG, computer vision, gaze detection, and partially autonomous guidance, which drastically improve the accuracy of online tasks and reduce the brain burden caused by long-term mental activities.

Neural Decoding of Chinese Sign Language With Machine Learning for Brain-Computer Interfaces

Limb motion decoding is an important part of brain-computer interface (BCI) research. Among the limb motion, sign language not only contains rich semantic information and abundant maneuverable actions but also provides different executable commands. However, many researchers focus on decoding the gross motor skills, such as the decoding of ordinary motor imagery or simple upper limb movements. Here we explored the neural features and decoding of Chinese sign language from electroencephalograph (EEG) signal with motor imagery and motor execution. Sign language not only contains rich semantic information, but also has abundant maneuverable actions, and provides us with more different executable commands. In this paper, twenty subjects were instructed to perform movement execution and movement imagery based on Chinese sign language. Seven classifiers are employed to classify the selected features of sign language EEG. L1 regularization is used to learn and select features that contain more information from the mean, power spectral density, sample entropy, and brain network connectivity. The best average classification accuracy of the classifier is 89.90% (imagery sign language is 83.40%). These results have shown the feasibility of decoding between different sign languages. The source location reveals that the neural circuits involved in sign language are related to the visual contact area and the pre-movement area. Experimental evaluation shows that the proposed decoding strategy based on sign language can obtain outstanding classification results, which provides a certain reference value for the subsequent research of limb decoding based on sign language.

Decoding Hand Movement Types and Kinematic Information From Electroencephalogram

Brain-computer interfaces (BCIs) have achieved successful control of assistive devices, e.g. neuroprosthesis or robotic arm. Previous research based on hand movements Electroencephalogram (EEG) has shown limited success in precise and natural control. In this study, we explored the possibilities of decoding movement types and kinematic information for three reach-and-execute actions using movement-related cortical potentials (MRCPs). EEG signals were acquired from 12 healthy subjects during the execution of pinch, palmar and precision disk rotation actions that involved two levels of speeds and forces. In the case of discrimination between hand movement types under each of four different kinematics conditions, we obtained the average peak accuracies of 83.44% and 73.83% for the binary and 3-class classification, respectively. In the case of discrimination between different movement kinematics for each of three actions, the average peak accuracies of 82.9% and 58.2% could be achieved for the two and 4-class scenario. In both cases, peak decoding performance was significantly higher than the subject-specific chance level. We found that hand movement types all could be classified when these actions were executed at four different kinematic parameters. Meanwhile, for each of three hand movements, we decoded movement parameters successfully. Furthermore, the feasibility of decoding hand movements during hand retraction process was also demonstrated. These findings are of great importance for controlling neuroprosthesis or other rehabilitation devices in a fine and natural way, which would drastically increase the acceptance of motor impaired users.

A Comprehensive Review on Critical Issues and Possible Solutions of Motor Imagery Based Electroencephalography Brain-Computer Interface

Motor imagery (MI) based brain–computer interface (BCI) aims to provide a means of communication through the utilization of neural activity generated due to kinesthetic imagination of limbs. Every year, a significant number of publications that are related to new improvements, challenges, and breakthrough in MI-BCI are made. This paper provides a comprehensive review of the electroencephalogram (EEG) based MI-BCI system. It describes the current state of the art in different stages of the MI-BCI (data acquisition, MI training, preprocessing, feature extraction, channel and feature selection, and classification) pipeline. Although MI-BCI research has been going for many years, this technology is mostly confined to controlled lab environments. We discuss recent developments and critical algorithmic issues in MI-based BCI for commercial deployment.