Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking

Intentional binding for noninvasive BCI control

Objective. Noninvasive brain-computer interfaces (BCIs) allow to interact with the external environment by naturally bypassing the musculoskeletal system. Making BCIs efficient and accurate is paramount to improve the reliability of real-life and clinical applications, from open-loop device control to closed-loop neurorehabilitation.Approach. By promoting sense of agency and embodiment, realistic setups including multimodal channels of communication, such as eye-gaze, and robotic prostheses aim to improve BCI performance. However, how the mental imagery command should be integrated in those hybrid systems so as to ensure the best interaction is still poorly understood. To address this question, we performed a hybrid EEG-based BCI training involving healthy volunteers enrolled in a reach-and-grasp action operated by a robotic arm.Main results. Showed that the hand grasping motor imagery timing significantly affects the BCI accuracy evolution as well as the spatiotemporal brain dynamics. Larger accuracy improvement was obtained when motor imagery is performed just after the robot reaching, as compared to before or during the movement. The proximity with the subsequent robot grasping favored intentional binding, led to stronger motor-related brain activity, and primed the ability of sensorimotor areas to integrate information from regions implicated in higher-order cognitive functions.Significance. Taken together, these findings provided fresh evidence about the effects of intentional binding on human behavior and cortical network dynamics that can be exploited to design a new generation of efficient brain-machine interfaces.

A novel feature extraction method PSS-CSP for binary motor imagery - based brain-computer interfaces

In order to improve the performance of binary motor imagery (MI) - based brain-computer interfaces (BCIs) using electroencephalography (EEG), a novel method (PSS-CSP) is proposed, which combines spectral subtraction with common spatial pattern. Spectral subtraction is an effective denoising method which is initially adopted to process MI-based EEG signals for binary BCIs in this work. On this basis, we proposed a novel feature extraction method called power spectral subtraction-based common spatial pattern (PSS-CSP) , which calculates the differences in power spectrum between binary classes of EEG signals and uses the differences in the feature extraction process. Additionally, support vector machine (SVM) algorithm is used for signal classification. Results show the proposed method (PSS-CSP) outperforms certain existing methods, achieving a classification accuracy of 76.8% on the BCIIV dataset 2b, and 76.25% and 77.38% on the OpenBMI dataset session 1 and session 2, respectively.

A high-performance general computer cursor control scheme based on a hybrid BCI combining motor imagery and eye-tracking

This study introduces a novel virtual cursor control system designed to empower individuals with neuromuscular disabilities in the digital world. By combining eye-tracking with motor imagery (MI) in a hybrid brain-computer interface (BCI), the system enhances cursor control accuracy and simplicity. Real-time classification accuracy reaches 87.92% (peak of 93.33%), with cursor stability in the gazing state at 96.1%. Integrated into common operating systems, it enables tasks like text entry, online chatting, email, web surfing, and picture dragging, with an average text input rate of 53.2 characters per minute (CPM). This technology facilitates fundamental computing tasks for patients, fostering their integration into the online community and paving the way for future developments in BCI systems.

A robust multi-branch multi-attention-mechanism EEGNet for motor imagery BCI decoding

BACKGROUND

Motor-Imagery-based Brain-Computer Interface (MI-BCI) is a promising technology to assist communication, movement, and neurological rehabilitation for motor-impaired individuals. Electroencephalography (EEG) decoding techniques using deep learning (DL) possess noteworthy advantages due to automatic feature extraction and end-to-end learning. However, the DL-based EEG decoding models tend to show large variations due to intersubject variability of EEG, which results from inconsistencies of different subjects' optimal hyperparameters.

NEW METHODS

This study proposes a multi-branch multi-attention mechanism EEGNet model (MBMANet) for robust decoding. It applies the multi-branch EEGNet structure to achieve various feature extractions. Further, the different attention mechanisms introduced in each branch attain diverse adaptive weight adjustments. This combination of multi-branch and multi-attention mechanisms allows for multi-level feature fusion to provide robust decoding for different subjects.

RESULTS

The MBMANet model has a four-classification accuracy of 83.18% and kappa of 0.776 on the BCI Competition IV-2a dataset, which outperforms other eight CNN-based decoding models. This consistently satisfactory performance across all nine subjects indicates that the proposed model is robust.

CONCLUSIONS

The combine of multi-branch and multi-attention mechanisms empowers the DL-based models to adaptively learn different EEG features, which provides a feasible solution for dealing with data variability. It also gives the MBMANet model more accurate decoding of motion intentions and lower training costs, thus improving the MI-BCI's utility and robustness.

Continuous Tracking using Deep Learning-based Decoding for Non-invasive Brain-Computer Interface

Brain-computer interfaces (BCI) using electroencephalography (EEG) provide a non-invasive method for users to interact with external devices without the need for muscle activation. While noninvasive BCIs have the potential to improve the quality of lives of healthy and motor impaired individuals, they currently have limited applications due to inconsistent performance and low degrees of freedom. In this study, we use deep learning (DL)-based decoders for online Continuous Pursuit (CP), a complex BCI task requiring the user to track an object in two-dimensional space. We developed a labeling system to use CP data for supervised learning, trained DL-based decoders based on two architectures, including a newly proposed adaptation of the PointNet architecture, and evaluated the performance over several online sessions. We rigorously evaluated the DL-based decoders in a total of 28 human participants, and found that the DL-based models improved throughout the sessions as more training data became available and significantly outperformed a traditional BCI decoder by the last session. We also performed additional experiments to test an implementation of transfer learning by pre-training models on data from other subjects, and mid-session training to reduce inter-session variability. The results from these experiments showed that pre-training did not significantly improve performance, but updating the models mid-session may have some benefit. Overall, these findings support the use of DL-based decoders for improving BCI performance in complex tasks like CP, which can expand the potential applications of BCI devices and help improve the quality of lives of healthy and motor-impaired individuals.

Significance Statement

Brain-computer Interfaces (BCI) have the potential to replace or restore motor functions for patients and can benefit the general population by providing a direct link of the brain with robotics or other devices. In this work, we developed a paradigm using deep learning (DL)-based decoders for continuous control of a BCI system and demonstrated its capabilities through extensive online experiments. We also investigate how DL performance is affected by varying amounts of training data and collected more than 150 hours of BCI data that can be used to train new models. The results of this study provide valuable information for developing future DL-based BCI decoders which can improve performance and help bring BCIs closer to practical applications and wide-spread use.

Continuous tracking using deep learning-based decoding for noninvasive brain-computer interface

Brain-computer interfaces (BCI) using electroencephalography provide a noninvasive method for users to interact with external devices without the need for muscle activation. While noninvasive BCIs have the potential to improve the quality of lives of healthy and motor-impaired individuals, they currently have limited applications due to inconsistent performance and low degrees of freedom. In this study, we use deep learning (DL)-based decoders for online continuous pursuit (CP), a complex BCI task requiring the user to track an object in 2D space. We developed a labeling system to use CP data for supervised learning, trained DL-based decoders based on two architectures, including a newly proposed adaptation of the PointNet architecture, and evaluated the performance over several online sessions. We rigorously evaluated the DL-based decoders in a total of 28 human participants, and found that the DL-based models improved throughout the sessions as more training data became available and significantly outperformed a traditional BCI decoder by the last session. We also performed additional experiments to test an implementation of transfer learning by pretraining models on data from other subjects, and midsession training to reduce intersession variability. The results from these experiments showed that pretraining did not significantly improve performance, but updating the models' midsession may have some benefit. Overall, these findings support the use of DL-based decoders for improving BCI performance in complex tasks like CP, which can expand the potential applications of BCI devices and help to improve the quality of lives of healthy and motor-impaired individuals.

Optimized FFNN with multichannel CSP-ICA framework of EEG signal for BCI

The electroencephalogram (EEG) of the patient is used to identify their motor intention, which is then converted into a control signal through a brain-computer interface (BCI) based on motor imagery. Whenever gathering features from EEG signals, making a BCI is difficult in part because of the enormous dimensionality of the data. Three stages make up the suggested methodology: pre-processing, extraction of features, selection, and categorization. To remove unwanted artifacts, the EEG signals are filtered by a fifth-order Butterworth multichannel band-pass filter. This decreases execution time and memory use, both of which improve system performance. Then a novel multichannel optimized CSP-ICA feature extraction technique is used to separate and eliminate non-discriminative information from discriminative information in the EEG channels. Furthermore, CSP uses the concept of an Artificial Bee Colony (ABC) algorithm to automatically identify the simultaneous global ideal frequency band and time interval combination for the extraction and classification of common spatial pattern characteristics. Finally, a Tunable optimized feed-forward neural network (FFNN) classifier is utilized to extract and categorize the temporal and frequency domain features, which employs an FFNN classifier with Tunable-Q wavelet transform. The proposed framework, therefore optimizes signal processing, enabling enhanced EEG signal classification for BCI applications. The result shows that the models that use Tunable optimized FFNN produce higher classification accuracy of more than 20% when compared to the existing models.

Personalized motor imagery prediction model based on individual difference of ERP

Objective. Motor imagery-based brain-computer interaction (MI-BCI) is a novel method of achieving human and external environment interaction that can assist individuals with motor disorders to rehabilitate. However, individual differences limit the utility of the MI-BCI. In this study, a personalized MI prediction model based on the individual difference of event-related potential (ERP) is proposed to solve the MI individual difference.Approach.A novel paradigm named action observation-based multi-delayed matching posture task evokes ERP during a delayed matching posture task phase by retrieving picture stimuli and videos, and generates MI electroencephalogram through action observation and autonomous imagery in an action observation-based motor imagery phase. Based on the correlation between the ERP and MI, a logistic regression-based personalized MI prediction model is built to predict each individual's suitable MI action. 32 subjects conducted the MI task with or without the help of the prediction model to select the MI action. Then classification accuracy of the MI task is used to evaluate the proposed model and three traditional MI methods.Main results.The personalized MI prediction model successfully predicts suitable action among 3 sets of daily actions. Under suitable MI action, the individual's ERP amplitude and event-related desynchronization (ERD) intensity are the largest, which helps to improve the accuracy by 14.25%.Significance.The personalized MI prediction model that uses the temporal ERP features to predict the classification accuracy of MI is feasible for improving the individual's MI-BCI performance, providing a new personalized solution for the individual difference and practical BCI application.

Advanced Electrode Technologies for Noninvasive Brain-Computer Interfaces

Brain-computer interfaces (BCIs) have garnered significant attention in recent years due to their potential applications in medical, assistive, and communication technologies. Building on this, noninvasive BCIs stand out as they provide a safe and user-friendly method for interacting with the human brain. In this work, we provide a comprehensive overview of the latest developments and advancements in material, design, and application of noninvasive BCIs electrode technology. We also explore the challenges and limitations currently faced by noninvasive BCI electrode technology and sketch out the technological roadmap from three dimensions: Materials and Design; Performances; Mode and Function. We aim to unite research efforts within the field of noninvasive BCI electrode technology, focusing on the consolidation of shared goals and fostering integrated development strategies among a diverse array of multidisciplinary researchers.

EEG-controlled tele-grasping for undefined objects

This paper presents a teleoperation system of robot grasping for undefined objects based on a real-time EEG (Electroencephalography) measurement and shared autonomy. When grasping an undefined object in an unstructured environment, real-time human decision is necessary since fully autonomous grasping may not handle uncertain situations. The proposed system allows involvement of a wide range of human decisions throughout the entire grasping procedure, including 3D movement of the gripper, selecting proper grasping posture, and adjusting the amount of grip force. These multiple decision-making procedures of the human operator have been implemented with six flickering blocks for steady-state visually evoked potentials (SSVEP) by dividing the grasping task into predefined substeps. Each substep consists of approaching the object, selecting posture and grip force, grasping, transporting to the desired position, and releasing. The graphical user interface (GUI) displays the current substep and simple symbols beside each flickering block for quick understanding. The tele-grasping of various objects by using real-time human decisions of selecting among four possible postures and three levels of grip force has been demonstrated. This system can be adapted to other sequential EEG-controlled teleoperation tasks that require complex human decisions.

Classification Algorithm for Electroencephalogram-based Motor Imagery Using Hybrid Neural Network with Spatio-temporal Convolution and Multi-head Attention Mechanism

Motor imagery (MI) is a brain-computer interface (BCI) technique in which specific brain regions are activated when people imagine their limbs (or muscles) moving, even without actual movement. The technology converts electroencephalogram (EEG) signals generated by the brain into computer-readable commands by measuring neural activity. Classification of motor imagery is one of the tasks in BCI. Researchers have done a lot of work on motor imagery classification, and the existing literature has relatively mature decoding methods for two-class motor tasks. However, as the categories of EEG-based motor imagery tasks increase, further exploration is needed for decoding research on four-class motor imagery tasks. In this study, we designed a hybrid neural network that combines spatiotemporal convolution and attention mechanisms. Specifically, the data is first processed by spatiotemporal convolution to extract features and then processed by a Multi-branch Convolution block. Finally, the processed data is input into the encoder layer of the Transformer for a self-attention calculation to obtain the classification results. Our approach was tested on the well-known MI datasets BCI Competition IV 2a and 2b, and the results show that the 2a dataset has a global average classification accuracy of 83.3% and a kappa value of 0.78. Experimental results show that the proposed method outperforms most of the existing methods.

MI-based BCI with accurate real-time three-class classification processing and light control application

The use of brain-computer interfaces (BCIs) to control intelligent devices is a current and future research direction. However, the challenges of low accuracy of real-time recognition and the need for multiple electroencephalographic channels are yet to be overcome. While a number of research teams have proposed many ways to improve offline classification accuracy, the potential problems in real-time experiments are often overlooked. In this study, we proposed a label-based channel diversion preprocessing to solve the problem of low real-time classification accuracy. The Tikhonov regularised common spatial-pattern algorithm (TRCSP) and one vs rest support vector machine (OVR-SVM) were used for feature extraction and pattern classification. High accuracy was achieved in real-time three-class classification using only three channels (average real-time accuracy of 87.46%, with a maximum of 90.33%). In addition, the stability and reliability of the system were verified through lighting control experiments in a real environment. Using the autonomy of MI and real-time feedback of light brightness, we have built a fully autonomous interactive system. The improvement in the real-time classification accuracy in this study is of great significance to the industrialisation of BCI.

State-of-the-Art on Brain-Computer Interface Technology

This paper provides a comprehensive overview of the state-of-the-art in brain-computer interfaces (BCI). It begins by providing an introduction to BCIs, describing their main operation principles and most widely used platforms. The paper then examines the various components of a BCI system, such as hardware, software, and signal processing algorithms. Finally, it looks at current trends in research related to BCI use for medical, educational, and other purposes, as well as potential future applications of this technology. The paper concludes by highlighting some key challenges that still need to be addressed before widespread adoption can occur. By presenting an up-to-date assessment of the state-of-the-art in BCI technology, this paper will provide valuable insight into where this field is heading in terms of progress and innovation.

A comprehensive review on motion trajectory reconstruction for EEG-based brain-computer interface

The advance in neuroscience and computer technology over the past decades have made brain-computer interface (BCI) a most promising area of neurorehabilitation and neurophysiology research. Limb motion decoding has gradually become a hot topic in the field of BCI. Decoding neural activity related to limb movement trajectory is considered to be of great help to the development of assistive and rehabilitation strategies for motor-impaired users. Although a variety of decoding methods have been proposed for limb trajectory reconstruction, there does not yet exist a review that covers the performance evaluation of these decoding methods. To alleviate this vacancy, in this paper, we evaluate EEG-based limb trajectory decoding methods regarding their advantages and disadvantages from a variety of perspectives. Specifically, we first introduce the differences in motor execution and motor imagery in limb trajectory reconstruction with different spaces (2D and 3D). Then, we discuss the limb motion trajectory reconstruction methods including experiment paradigm, EEG pre-processing, feature extraction and selection, decoding methods, and result evaluation. Finally, we expound on the open problem and future outlooks.

Sensor Fusion-Based Teleoperation Control of Anthropomorphic Robotic Arm

Sensor fusion is a technique that combines information from multiple sensors in order to improve the accuracy and reliability of the data being collected. In the context of teleoperation control of an anthropomorphic robotic arm, sensor fusion technology can be used to enhance the precise control of anthropomorphic robotic arms by combining data from multiple sensors, such as cameras, data gloves, force sensors, etc. By fusing and processing this sensing information, it can enable real-time control of anthropomorphic robotic arms and dexterous hands, replicating the motion of human manipulators. In this paper, we present a sensor fusion-based teleoperation control system for the anthropomorphic robotic arm and dexterous hand, which utilizes a filter to fuse data from multiple sensors in real-time. As such, the real-time perceived human arms motion posture information is analyzed and processed, and wireless communication is used to intelligently and flexibly control the anthropomorphic robotic arm and dexterous hand. Finally, the user is able to manage the anthropomorphic operation function in a stable and reliable manner. We also discussed the implementation and experimental evaluation of the system, showing that it is able to achieve improved performance and stability compared to traditional teleoperation control methods.

Advanced Modeling and Signal Processing Methods in Brain-Computer Interfaces Based on a Vector of Cyclic Rhythmically Connected Random Processes

In this study is substantiated the new mathematical model of vector of electroencephalographic signals, registered under the conditions of multiple repetitions of the mental control influences of brain-computer interface operator, in the form of a vector of cyclic rhythmically connected random processes, which, due to taking into account the stochasticity and cyclicity, the variability and commonality of the rhythm of the investigated signals have a number of advantages over the known models. This new model opens the way for the study of multidimensional distribution functions; initial, central, and mixed moment functions of higher order such as for each electroencephalographic signal separately; as well as for their respective compatible probabilistic characteristics, among which the most informative characteristics can be selected. This provides an increase in accuracy in the detection (classification) of mental control influences of the brain-computer interface operators. Based on the developed mathematical model, the statistical processing methods of vector of electroencephalographic signals are substantiated, which consist of statistical evaluation of its probabilistic characteristics and make it possible to conduct an effective joint statistical estimation of the probability characteristics of electroencephalographic signals. This provides the basis for coordinated integration of information from different sensors. The use of moment functions of higher order and their spectral images in the frequency domain, as informative characteristics in brain-computer interface systems, are substantiated. Their significant sensitivity to the mental controlling influence of the brain-computer interface operator is experimentally established. The application of Bessel's inequality to the problems of reducing the dimensions (from 500 to 20 numbers) of the vectors of informative features makes it possible to significantly reduce the computational complexity of the algorithms for the functioning of brain-computer interface systems. Namely, we experimentally established that only the first 20 values of the Fourier transform of the estimation of moment functions of higher-order electroencephalographic signals are sufficient to form the vector of informative features in brain-computer interface systems, because these spectral components make up at least 95% of the total energy of the corresponding statistical estimate of the moment functions of higher-order electroencephalographic signals.

Subject-Dependent Artifact Removal for Enhancing Motor Imagery Classifier Performance under Poor Skills

The Electroencephalography (EEG)-based motor imagery (MI) paradigm is one of the most studied technologies for Brain-Computer Interface (BCI) development. Still, the low Signal-to-Noise Ratio (SNR) poses a challenge when constructing EEG-based BCI systems. Moreover, the non-stationary and nonlinear signal issues, the low-spatial data resolution, and the inter- and intra-subject variability hamper the extraction of discriminant features. Indeed, subjects with poor motor skills have difficulties in practicing MI tasks against low SNR scenarios. Here, we propose a subject-dependent preprocessing approach that includes the well-known Surface Laplacian Filtering and Independent Component Analysis algorithms to remove signal artifacts based on the MI performance. In addition, power- and phase-based functional connectivity measures are studied to extract relevant and interpretable patterns and identify subjects of inefficency. As a result, our proposal, Subject-dependent Artifact Removal (SD-AR), improves the MI classification performance in subjects with poor motor skills. Consequently, electrooculography and volume-conduction EEG artifacts are mitigated within a functional connectivity feature-extraction strategy, which favors the classification performance of a straightforward linear classifier.

EEG-Based Empathic Safe Cobot

An empathic collaborative robot (cobot) was realized through the transmission of fear from a human agent to a robot agent. Such empathy was induced through an electroencephalographic (EEG) sensor worn by the human agent, thus realizing an empathic safe brain-computer interface (BCI). The empathic safe cobot reacts to the fear and in turn transmits it to the human agent, forming a social circle of empathy and safety. A first randomized, controlled experiment involved two groups of 50 healthy subjects (100 total subjects) to measure the EEG signal in the presence or absence of a frightening event. The second randomized, controlled experiment on two groups of 50 different healthy subjects (100 total subjects) exposed the subjects to comfortable and uncomfortable movements of a collaborative robot (cobot) while the subjects’ EEG signal was acquired. The result was that a spike in the subject’s EEG signal was observed in the presence of uncomfortable movement. The questionnaires were distributed to the subjects, and confirmed the results of the EEG signal measurement. In a controlled laboratory setting, all experiments were found to be statistically significant. In the first experiment, the peak EEG signal measured just after the activating event was greater than the resting EEG signal (p < 10−3). In the second experiment, the peak EEG signal measured just after the uncomfortable movement of the cobot was greater than the EEG signal measured under conditions of comfortable movement of the cobot (p < 10−3). In conclusion, within the isolated and constrained experimental environment, the results were satisfactory.

Manual 3D Control of an Assistive Robotic Manipulator Using Alpha Rhythms and an Auditory Menu: A Proof-of-Concept

Brain–Computer Interfaces (BCIs) have been regarded as potential tools for individuals with severe motor disabilities, such as those with amyotrophic lateral sclerosis, that render interfaces that rely on movement unusable. This study aims to develop a dependent BCI system for manual end-point control of a robotic arm. A proof-of-concept system was devised using parieto-occipital alpha wave modulation and a cyclic menu with auditory cues. Users choose a movement to be executed and asynchronously stop said action when necessary. Tolerance intervals allowed users to cancel or confirm actions. Eight able-bodied subjects used the system to perform a pick-and-place task. To investigate the potential learning effects, the experiment was conducted twice over the course of two consecutive days. Subjects obtained satisfactory completion rates (84.0 ± 15.0% and 74.4 ± 34.5% for the first and second day, respectively) and high path efficiency (88.9 ± 11.7% and 92.2 ± 9.6%). Subjects took on average 439.7 ± 203.3 s to complete each task, but the robot was only in motion 10% of the time. There was no significant difference in performance between both days. The developed control scheme provided users with intuitive control, but a considerable amount of time is spent waiting for the right target (auditory cue). Implementing other brain signals may increase its speed.

Augmented Reality-Based Interface for Bimanual Robot Teleoperation

Teleoperation of bimanual robots is being used to carry out complex tasks such as surgeries in medicine. Despite the technological advances, current interfaces are not natural to the users, who spend long periods of time in learning how to use these interfaces. In order to mitigate this issue, this work proposes a novel augmented reality-based interface for teleoperating bimanual robots. The proposed interface is more natural to the user and reduces the interface learning process. A full description of the proposed interface is detailed in the paper, whereas its effectiveness is shown experimentally using two industrial robot manipulators. Moreover, the drawbacks and limitations of the classic teleoperation interface using joysticks are analyzed in order to highlight the benefits of the proposed augmented reality-based interface approach.