BCI-controlled wheelchairs: end-users' perceptions, needs, and expectations, an interview-based study

PURPOSE

Brain-computer interface (BCI)-controlled wheelchairs have the potential to improve the independence of people with mobility impairments. The low uptake of BCI devices has been linked to a lack of knowledge among researchers of the needs of end-users that should influence BCI development.

MATERIALS AND METHODS

This study used semi-structured interviews to learn about the perceptions, needs, and expectations of spinal cord injury (SCI) patients with regards to a BCI-controlled wheelchair. Topics discussed in the interview include: paradigms, shared control, safety, robustness, channel selection, hardware, and experimental design. The interviews were recorded and then transcribed. Analysis was carried out using coding based on grounded theory principles.

RESULTS

The majority of participants had a positive view of BCI-controlled wheelchair technology and were willing to use the technology. Core issues were raised regarding safety, cost and aesthetics. Interview discussions were linked to state-of-the-art BCI technology. The results challenge the current reliance of researchers on the motor-imagery paradigm by suggesting end-users expect highly intuitive paradigms. There also needs to be a stronger focus on obstacle avoidance and safety features in BCI wheelchairs. Finally, the development of control approaches that can be personalized for individual users may be instrumental for widespread adoption of these devices.

CONCLUSIONS

This study, based on interviews with SCI patients, indicates that BCI-controlled wheelchairs are a promising assistive technology that would be well received by end-users. Recommendations for a more person-centered design of BCI controlled wheelchairs are made and clear avenues for future research are identified.

Transfer learning promotes acquisition of individual BCI skills

Subject training is crucial for acquiring brain-computer interface (BCI) control. Typically, this requires collecting user-specific calibration data due to high inter-subject neural variability that limits the usability of generic decoders. However, calibration is cumbersome and may produce inadequate data for building decoders, especially with naïve subjects. Here, we show that a decoder trained on the data of a single expert is readily transferrable to inexperienced users via domain adaptation techniques allowing calibration-free BCI training. We introduce two real-time frameworks, (i) Generic Recentering (GR) through unsupervised adaptation and (ii) Personally Assisted Recentering (PAR) that extends GR by employing supervised recalibration of the decoder parameters. We evaluated our frameworks on 18 healthy naïve subjects over five online sessions, who operated a customary synchronous bar task with continuous feedback and a more challenging car racing game with asynchronous control and discrete feedback. We show that along with improved task-oriented BCI performance in both tasks, our frameworks promoted subjects' ability to acquire individual BCI skills, as the initial neurophysiological control features of an expert subject evolved and became subject specific. Furthermore, those features were task-specific and were learned in parallel as participants practiced the two tasks in every session. Contrary to previous findings implying that supervised methods lead to improved online BCI control, we observed that longitudinal training coupled with unsupervised domain matching (GR) achieved similar performance to supervised recalibration (PAR). Therefore, our presented frameworks facilitate calibration-free BCIs and have immediate implications for broader populations-such as patients with neurological pathologies-who might struggle to provide suitable initial calibration data.

Continuous shared control of a mobile robot with brain-computer interface and autonomous navigation for daily assistance

Although the electroencephalography (EEG) based brain-computer interface (BCI) has been successfully developed for rehabilitation and assistance, it is still challenging to achieve continuous control of a brain-actuated mobile robot system. In this study, we propose a continuous shared control strategy combining continuous BCI and autonomous navigation for a mobile robot system. The weight of shared control is designed to dynamically adjust the fusion of continuous BCI control and autonomous navigation. During this process, the system uses the visual-based simultaneous localization and mapping (SLAM) method to construct environmental maps. After obtaining the global optimal path, the system utilizes the brain-based shared control dynamic window approach (BSC-DWA) to evaluate safe and reachable trajectories while considering shared control velocity. Eight subjects participated in two-stage training, and six of these eight subjects participated in online shared control experiments. The training results demonstrated that naïve subjects could achieve continuous control performance with an average percent valid correct rate of approximately 97 % and an average total correct rate of over 80 %. The results of online shared control experiments showed that all of the subjects could complete navigation tasks in an unknown corridor with continuous shared control. Therefore, our experiments verified the feasibility and effectiveness of the proposed system combining continuous BCI, shared control, autonomous navigation, and visual SLAM. The proposed continuous shared control framework shows great promise in BCI-driven tasks, especially navigation tasks for brain-driven assistive mobile robots and wheelchairs in daily applications.

Electroencephalogram based brain-computer interface: Applications, challenges, and opportunities

Brain-Computer Interfaces (BCI) is an exciting and emerging research area for researchers and scientists. It is a suitable combination of software and hardware to operate any device mentally. This review emphasizes the significant stages in the BCI domain, current problems, and state-of-the-art findings. This article also covers how current results can contribute to new knowledge about BCI, an overview of BCI from its early developments to recent advancements, BCI applications, challenges, and future directions. The authors pointed to unresolved issues and expressed how BCI is valuable for analyzing the human brain. Humans' dependence on machines has led humankind into a new future where BCI can play an essential role in improving this modern world.

Teleoperation control of a wheeled mobile robot based on Brain-machine Interface

This paper presents a novel teleoperation system using Electroencephalogram (EEG) to control the motion of a wheeled mobile robot (WMR). Different from the other traditional motion controlling method, the WMR is braked with the EEG classification results. Furthermore, the EEG will be induced by using the online BMI (Brain Machine Interface) system, and adopting the non-intrusion induced mode SSVEP (steady state visually evoked potentials). Then, user's motion intention can be recognized by canonical correlation analysis (CCA) classifier, which will be converted into motion commands of the WMR. Finally, the teleoperation technique is utilized to manage the information of the movement scene and adjust the control instructions based on the real-time information. Bezier curve is used to parameterize the path planning of the robot, and the trajectory can be adjusted in real time by EEG recognition results. A motion controller based on error model is proposed to track the planned trajectory by using velocity feedback control, providing excellent track tracking performance. Finally, the feasibility and performance of the proposed teleoperation brain-controlled WMR system are verified using demonstration experiments.

Learning to control a BMI-driven wheelchair for people with severe tetraplegia

Mind-controlled wheelchairs are an intriguing assistive mobility solution applicable in complete paralysis. Despite progress in brain-machine interface (BMI) technology, its translation remains elusive. The primary objective of this study is to probe the hypothesis that BMI skill acquisition by end-users is fundamental to control a non-invasive brain-actuated intelligent wheelchair in real-world settings. We demonstrate that three tetraplegic spinal-cord injury users could be trained to operate a non-invasive, self-paced thought-controlled wheelchair and execute complex navigation tasks. However, only the two users exhibiting increasing decoding performance and feature discriminancy, significant neuroplasticity changes and improved BMI command latency, achieved high navigation performance. In addition, we show that dexterous, continuous control of robots is possible through low-degree of freedom, discrete and uncertain control channels like a motor imagery BMI, by blending human and artificial intelligence through shared-control methodologies. We posit that subject learning and shared-control are the key components paving the way for translational non-invasive BMI.

A Comprehensive Review of Endogenous EEG-Based BCIs for Dynamic Device Control

Electroencephalogram (EEG)-based brain–computer interfaces (BCIs) provide a novel approach for controlling external devices. BCI technologies can be important enabling technologies for people with severe mobility impairment. Endogenous paradigms, which depend on user-generated commands and do not need external stimuli, can provide intuitive control of external devices. This paper discusses BCIs to control various physical devices such as exoskeletons, wheelchairs, mobile robots, and robotic arms. These technologies must be able to navigate complex environments or execute fine motor movements. Brain control of these devices presents an intricate research problem that merges signal processing and classification techniques with control theory. In particular, obtaining strong classification performance for endogenous BCIs is challenging, and EEG decoder output signals can be unstable. These issues present myriad research questions that are discussed in this review paper. This review covers papers published until the end of 2021 that presented BCI-controlled dynamic devices. It discusses the devices controlled, EEG paradigms, shared control, stabilization of the EEG signal, traditional machine learning and deep learning techniques, and user experience. The paper concludes with a discussion of open questions and avenues for future work.

Neural correlates of user learning during long-term BCI training for the Cybathlon competition

Background

Brain-computer interfaces (BCIs) are systems capable of translating human brain patterns, measured through electroencephalography (EEG), into commands for an external device. Despite the great advances in machine learning solutions to enhance the performance of BCI decoders, the translational impact of this technology remains elusive. The reliability of BCIs is often unsatisfactory for end-users, limiting their application outside a laboratory environment.

Methods

We present the analysis on the data acquired from an end-user during the preparation for two Cybathlon competitions, where our pilot won the gold medal twice in a row. These data are of particular interest given the mutual learning approach adopted during the longitudinal training phase (8 months), the long training break in between the two events (1 year) and the demanding evaluation scenario. A multifaceted perspective on long-term user learning is proposed: we enriched the information gathered through conventional metrics (e.g., accuracy, application performances) by investigating novel neural correlates of learning in different neural domains.

Results

First, we showed that by focusing the training on user learning, the pilot was capable of significantly improving his performance over time even with infrequent decoder re-calibrations. Second, we revealed that the analysis of the within-class modifications of the pilot’s neural patterns in the Riemannian domain is more effective in tracking the acquisition and the stabilization of BCI skills, especially after the 1-year break. These results further confirmed the key role of mutual learning in the acquisition of BCI skills, and particularly highlighted the importance of user learning as a key to enhance BCI reliability.

Conclusion

We firmly believe that our work may open new perspectives and fuel discussions in the BCI field to shift the focus of future research: not only to the machine learning of the decoder, but also in investigating novel training procedures to boost the user learning and the stability of the BCI skills in the long-term. To this end, the analyses and the metrics proposed could be used to monitor the user learning during training and provide a marker guiding the decoder re-calibration to maximize the mutual adaptation of the user to the BCI system.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12984-022-01047-x.

Sliding-Mode Nonlinear Predictive Control of Brain-Controlled Mobile Robots

In this article, we develop a robust sliding-mode nonlinear predictive controller for brain-controlled robots with enhanced performance, safety, and robustness. First, the kinematics and dynamics of a mobile robot are built. After that, the proposed controller is developed by cascading a predictive controller and a smooth sliding-mode controller. The predictive controller integrates the human intention tracking with safety guarantee objectives into an optimization problem to minimize the invasion to human intention while maintaining robot safety. The smooth sliding-mode controller is designed to achieve robust desired velocity tracking. The results of human-in-the-loop simulation and robotic experiments both show the efficacy and robust performance of the proposed controller. This work provides an enabling design to enhance the future research and development of brain-controlled robots.

ROS-Neuro: An Open-Source Platform for Neurorobotics

The growing interest in neurorobotics has led to a proliferation of heterogeneous neurophysiological-based applications controlling a variety of robotic devices. Although recent years have seen great advances in this technology, the integration between human neural interfaces and robotics is still limited, making evident the necessity of creating a standardized research framework bridging the gap between neuroscience and robotics. This perspective paper presents Robot Operating System (ROS)-Neuro, an open-source framework for neurorobotic applications based on ROS. ROS-Neuro aims to facilitate the software distribution, the repeatability of the experimental results, and support the birth of a new community focused on neuro-driven robotics. In addition, the exploitation of Robot Operating System (ROS) infrastructure guarantees stability, reliability, and robustness, which represent fundamental aspects to enhance the translational impact of this technology. We suggest that ROS-Neuro might be the future development platform for the flourishing of a new generation of neurorobots to promote the rehabilitation, the inclusion, and the independence of people with disabilities in their everyday life.

Continuous Hybrid BCI Control for Robotic Arm Using Noninvasive Electroencephalogram, Computer Vision, and Eye Tracking

The controlling of robotic arms based on brain–computer interface (BCI) can revolutionize the quality of life and living conditions for individuals with physical disabilities. Invasive electroencephalography (EEG)-based BCI has been able to control multiple degrees of freedom (DOFs) robotic arms in three dimensions. However, it is still hard to control a multi-DOF robotic arm to reach and grasp the desired target accurately in complex three-dimensional (3D) space by a noninvasive system mainly due to the limitation of EEG decoding performance. In this study, we propose a noninvasive EEG-based BCI for a robotic arm control system that enables users to complete multitarget reach and grasp tasks and avoid obstacles by hybrid control. The results obtained from seven subjects demonstrated that motor imagery (MI) training could modulate brain rhythms, and six of them completed the online tasks using the hybrid-control-based robotic arm system. The proposed system shows effective performance due to the combination of MI-based EEG, computer vision, gaze detection, and partially autonomous guidance, which drastically improve the accuracy of online tasks and reduce the brain burden caused by long-term mental activities.

Effects of Gaze Fixation on the Performance of a Motor Imagery-Based Brain-Computer Interface

Motor imagery-based brain-computer interfaces (BCIs) have been studied without controlling subjects’ gaze fixation position previously. The effect of gaze fixation and covert attention on the behavioral performance of BCI is still unknown. This study designed a gaze fixation controlled experiment. Subjects were required to conduct a secondary task of gaze fixation when performing the primary task of motor imagination. Subjects’ performance was analyzed according to the relationship between motor imagery target and the gaze fixation position, resulting in three BCI control conditions, i.e., congruent, incongruent, and center cross trials. A group of fourteen subjects was recruited. The average group performances of three different conditions did not show statistically significant differences in terms of BCI control accuracy, feedback duration, and trajectory length. Further analysis of gaze shift response time revealed a significantly shorter response time for congruent trials compared to incongruent trials. Meanwhile, the parietal occipital cortex also showed active neural activities for congruent and incongruent trials, and this was revealed by a contrast analysis of R-square values and lateralization index. However, the lateralization index computed from the parietal and occipital areas was not correlated with the BCI behavioral performance. Subjects’ BCI behavioral performance was not affected by the position of gaze fixation and covert attention. This indicated that motor imagery-based BCI could be used freely in robotic arm control without sacrificing performance.

Progress in Brain Computer Interface: Challenges and Opportunities

Brain computer interfaces (BCI) provide a direct communication link between the brain and a computer or other external devices. They offer an extended degree of freedom either by strengthening or by substituting human peripheral working capacity and have potential applications in various fields such as rehabilitation, affective computing, robotics, gaming, and neuroscience. Significant research efforts on a global scale have delivered common platforms for technology standardization and help tackle highly complex and non-linear brain dynamics and related feature extraction and classification challenges. Time-variant psycho-neurophysiological fluctuations and their impact on brain signals impose another challenge for BCI researchers to transform the technology from laboratory experiments to plug-and-play daily life. This review summarizes state-of-the-art progress in the BCI field over the last decades and highlights critical challenges.

A review of artificial intelligence for EEG‐based brain−computer interfaces and applications

The advancement in neuroscience and computer science promotes the ability of the human brain to communicate and interact with the environment, making brain–computer interface (BCI) top interdisciplinary research. Furthermore, with the modern technology advancement in artificial intelligence (AI), including machine learning (ML) and deep learning (DL) methods, there is vast growing interest in the electroencephalogram (EEG)‐based BCIs for AI‐related visual, literal, and motion applications. In this review study, the literature on mainstreams of AI for the EEG‐based BCI applications is investigated to fill gaps in the interdisciplinary BCI field. Specifically, the EEG signals and their main applications in BCI are first briefly introduced. Next, the latest AI technologies, including the ML and DL models, are presented to monitor and feedback human cognitive states. Finally, some BCI‐inspired AI applications, including computer vision, natural language processing, and robotic control applications, are presented. The future research directions of the EEG‐based BCI are highlighted in line with the AI technologies and applications.

Modeling of Human Operator Behavior for Brain-Actuated Mobile Robots Steering

Human operator control of brain-actuated robot steering based on electroencephalograph (EEG)-signals is a complex behavior consisting of surroundings perceiving, decision making, and commands issuing and differs among individual operators. However, no existing models allow decoupling the user from the loop to improve the system design and testing process, which can capture such behavior of a brain-actuated robot. To address this problem, in this paper, we propose an operator brain-controlled steering model consisting of an operator decision model based on the queuing network (QN) cognitive architecture and a brain-machine interface (BMI) performance model. The QN-based operator decision model can mimic the human decision process with the individual operator differences considered. The new BMI performance model is built to represent the varied accuracy of BMI during brain-controlled direction operations. Furthermore, the model is simulated and validated against the results of human operator-in-the-loop experiments. The results show that the proposed model can reproduce the behavior of human operators thanks to its similar direction control performance.