UMA-BCI Speller: An easily configurable P300 speller tool for end users

BACKGROUND AND OBJECTIVE

Some neurodegenerative conditions can severely limit patients' capability to communicate because of the loss of muscular control. Brain-computer interfaces may help in the restoration of communication with these patients, bypassing the muscular activity, so that brain signals can be directly interpreted by a computer. There are many studies regarding brain-controlled spellers; however, these systems do not usually leap out of the lab because of technical and economic requirements. As a consequence, the potential end users do not benefit from these scientific advances in their daily life. The objective of this paper is to present a novel brain-controlled speller designed to be used by patients due to its versatility and ease of use.

METHODS

The brain-computer interface research group of the University of Málaga (UMA-BCI) has developed a speller application based on the well-known P300 potential which can be easily installed, configured and used. The application supports the common P300 paradigms: the Row-Column Paradigm and the Rapid Serial Visual Presentation Paradigm. The inner core of the application is implemented with a widely used and studied platform, BCI2000, which ensures its reliability and allows other researchers to apply modifications at will in order to test new features. Ten naïve volunteers carried out exercises using the application and completed usability tests for evaluation purposes.

RESULTS

New subjects using the application managed to set up and use the proposed speller in less than an hour. The positive results of the evaluation through the usability tests support this application's ease of use.

CONCLUSIONS

A new brain-controlled spelling tool has been presented whose aim is to be used by severely paralyzed patients in their daily lives, as well as by researchers to test new spelling features.

Design, analysis and performance evaluation of a hand gesture platform for navigation

Prevailing technological solutions that address the problems that are experienced by the infirm and elderly people in terms of locomotion needs, offer limited options when it comes to control mechanism and customization. For more than a decade, joysticks have served the purpose of steering and navigation of autonomous wheelchairs. An alternative gesture-based method for navigation of wheelchairs by the physically impaired could very well replace the conventional joystick controls. A prototype system, 'Mudra' (Sanskrit word for gesture), incorporates a gesture capture module, developed for recognition and identification of hand gestures. Mudra is a no-nonsense user-friendly system that enables control of the navigational platform, merely by four gestures of the human hand. This paper presents a comprehensive report on the implementation of the Mudra system's hardware and software, performance analysis and safety evaluation. Verification tests of the conceptual design show promising results, where 97.8% of the hand gestures were recognized accurately. Furthermore, the response timings of wheelchairs with Mudra controls were noticeably faster than the joystick-based wheelchairs, as affirmed by live testing with wheelchair-users. Pursuant to the positive feedback from the wheelchair-user experience, we conclude that Mudra's gesture controlled wheelchairs would be a preferable alternative to joystick-controlled wheelchairs.

Implantable Neural Probes for Brain-Machine Interfaces - Current Developments and Future Prospects

A Brain-Machine interface (BMI) allows for direct communication between the brain and machines. Neural probes for recording neural signals are among the essential components of a BMI system. In this report, we review research regarding implantable neural probes and their applications to BMIs. We first discuss conventional neural probes such as the tetrode, Utah array, Michigan probe, and electroencephalography (ECoG), following which we cover advancements in next-generation neural probes. These next-generation probes are associated with improvements in electrical properties, mechanical durability, biocompatibility, and offer a high degree of freedom in practical settings. Specifically, we focus on three key topics: (1) novel implantable neural probes that decrease the level of invasiveness without sacrificing performance, (2) multi-modal neural probes that measure both electrical and optical signals, (3) and neural probes developed using advanced materials. Because safety and precision are critical for practical applications of BMI systems, future studies should aim to enhance these properties when developing next-generation neural probes.

A review of disability EEG based wheelchair control system: Coherent taxonomy, open challenges and recommendations

CONTEXT

Intelligent wheelchair technology has recently been utilised to address several mobility problems. Techniques based on brain-computer interface (BCI) are currently used to develop electric wheelchairs. Using human brain control in wheelchairs for people with disability has elicited widespread attention due to its flexibility.

OBJECTIVE

This study aims to determine the background of recent studies on wheelchair control based on BCI for disability and map the literature survey into a coherent taxonomy. The study intends to identify the most important aspects in this emerging field as an impetus for using BCI for disability in electric-powered wheelchair (EPW) control, which remains a challenge. The study also attempts to provide recommendations for solving other existing limitations and challenges.

METHODS

We systematically searched all articles about EPW control based on BCI for disability in three popular databases: ScienceDirect, IEEE and Web of Science. These databases contain numerous articles that considerably influenced this field and cover most of the relevant theoretical and technical issues.

RESULTS

We selected 100 articles on the basis of our inclusion and exclusion criteria. A large set of articles (55) discussed on developing real-time wheelchair control systems based on BCI for disability signals. Another set of articles (25) focused on analysing BCI for disability signals for wheelchair control. The third set of articles (14) considered the simulation of wheelchair control based on BCI for disability signals. Four articles designed a framework for wheelchair control based on BCI for disability signals. Finally, one article reviewed concerns regarding wheelchair control based on BCI for disability signals.

DISCUSSION

Since 2007, researchers have pursued the possibility of using BCI for disability in EPW control through different approaches. Regardless of type, articles have focused on addressing limitations that impede the full efficiency of BCI for disability and recommended solutions for these limitations.

CONCLUSIONS

Studies on wheelchair control based on BCI for disability considerably influence society due to the large number of people with disability. Therefore, we aim to provide researchers and developers with a clear understanding of this platform and highlight the challenges and gaps in the current and future studies.

Towards BCI-actuated smart wheelchair system

Background

Electroencephalogram-based brain–computer interfaces (BCIs) represent novel human machine interactive technology that allows people to communicate and interact with the external world without relying on their peripheral muscles and nervous system. Among BCI systems, brain-actuated wheelchairs are promising systems for the rehabilitation of severely motor disabled individuals who are unable to control a wheelchair by conventional interfaces. Previous related studies realized the easy use of brain-actuated wheelchairs that enable people to navigate the wheelchair through simple commands; however, these systems rely on offline calibration of the environment. Other systems do not rely on any prior knowledge; however, the control of the system is time consuming. In this paper, we have proposed an improved mobile platform structure equipped with an omnidirectional wheelchair, a lightweight robotic arm, a target recognition module and an auto-control module. Based on the you only look once (YOLO) algorithm, our system can, in real time, recognize and locate the targets in the environment, and the users confirm one target through a P300-based BCI. An expert system plans a proper solution for a specific target; for example, the planned solution for a door is opening the door and then passing through it, and the auto-control system then jointly controls the wheelchair and robotic arm to complete the operation. During the task execution, the target is also tracked by using an image tracking technique. Thus, we have formed an easy-to-use system that can provide accurate services to satisfy user requirements, and this system can accommodate different environments.

Results

To validate and evaluate our system, an experiment simulating the daily application was performed. The tasks included the user driving the system closer to a walking man and having a conversation with him; going to another room through a door; and picking up a bottle of water on the desk and drinking water. Three patients (cerebral infarction; spinal injury; and stroke) and four healthy subjects participated in the test and all completed the tasks.

Conclusion

This article presents a brain-actuated smart wheelchair system. The system is intelligent in that it provides efficient and considerate services for users. To test the system, three patients and four healthy subjects were recruited to participate in a test. The results demonstrate that the system works smartly and efficiently; with this system, users only need to issue small commands to get considerate services. This system is of significance for accelerating the application of BCIs in the practical environment, especially for patients who will use a BCI for rehabilitation applications.

Self-Paced Operation of a Wheelchair Based on a Hybrid Brain-Computer Interface Combining Motor Imagery and P300 Potential

This paper presents a hybrid brain-computer interface (BCI) that combines motor imagery (MI) and P300 potential for the asynchronous operation of a brain-controlled wheelchair whose design is based on a Mecanum wheel. This paradigm is completely user-centric. By sequentially performing MI tasks or paying attention to P300 flashing, the user can use eleven functions to control the wheelchair: move forward/backward, move left/right, move left45/right45, accelerate/decelerate, turn left/right, and stop. The practicality and effectiveness of the proposed approach were validated in eight subjects, all of whom achieved good performance. The preliminary results indicated that the proposed hybrid BCI system with different mental strategies operating sequentially is feasible and has potential applications for practical self-paced control.

The control of a virtual automatic car based on multiple patterns of motor imagery BCI

Multiple degrees of freedom (DOF) commands are required for a brain-actuated virtual automatic car, which makes the brain-computer interface (BCI) control strategy a big challenge. In order to solve the challenging issue, a mixed model of BCI combining P300 potentials and motor imagery had been realized in our previous study. However, compared with single model BCI, more training procedures are needed for the mixed model and more mental workload for users to bear. In the present study, we propose a multiple patterns of motor imagery (MPMI) BCI method, which is based on the traditional two patterns of motor imagery. Our motor imagery BCI approach had been extended to multiple patterns: right-hand motor imagery, left-hand motor imagery, foot motor imagery, and both hands motor imagery resulting in turning right, turning left, acceleration, and deceleration for a virtual automatic car control. Ten healthy subjects participated in online experiments, the experimental results not only show the efficiency of our proposed MPMI-BCI strategy but also indicate that those users can control the virtual automatic car spontaneously and efficiently without any other visual attention. Furthermore, the metric of path length optimality ratio (1.23) is very encouraging and the time optimality ratio (1.28) is especially remarkable. Graphical Abstract The paradigm of multiple patterns of motor imagery detection and the relevant topographies of CSP weights for different MI patterns.

EEG-Based BCI Control Schemes for Lower-Limb Assistive-Robots

Over recent years, brain-computer interface (BCI) has emerged as an alternative communication system between the human brain and an output device. Deciphered intents, after detecting electrical signals from the human scalp, are translated into control commands used to operate external devices, computer displays and virtual objects in the real-time. BCI provides an augmentative communication by creating a muscle-free channel between the brain and the output devices, primarily for subjects having neuromotor disorders, or trauma to nervous system, notably spinal cord injuries (SCI), and subjects with unaffected sensorimotor functions but disarticulated or amputated residual limbs. This review identifies the potentials of electroencephalography (EEG) based BCI applications for locomotion and mobility rehabilitation. Patients could benefit from its advancements such as wearable lower-limb (LL) exoskeletons, orthosis, prosthesis, wheelchairs, and assistive-robot devices. The EEG communication signals employed by the aforementioned applications that also provide feasibility for future development in the field are sensorimotor rhythms (SMR), event-related potentials (ERP) and visual evoked potentials (VEP). The review is an effort to progress the development of user's mental task related to LL for BCI reliability and confidence measures. As a novel contribution, the reviewed BCI control paradigms for wearable LL and assistive-robots are presented by a general control framework fitting in hierarchical layers. It reflects informatic interactions, between the user, the BCI operator, the shared controller, the robotic device and the environment. Each sub layer of the BCI operator is discussed in detail, highlighting the feature extraction, classification and execution methods employed by the various systems. All applications' key features and their interaction with the environment are reviewed for the EEG-based activity mode recognition, and presented in form of a table. It is suggested to structure EEG-BCI controlled LL assistive devices within the presented framework, for future generation of intent-based multifunctional controllers. Despite the development of controllers, for BCI-based wearable or assistive devices that can seamlessly integrate user intent, practical challenges associated with such systems exist and have been discerned, which can be constructive for future developments in the field.

Evaluation of Switch and Continuous Navigation Paradigms to Command a Brain-Controlled Wheelchair

A brain-computer interface (BCI) is a technology allowing patients with severe motor dysfunctions to use their electroencephalographic signals to create a communication channel to control devices. The objective of this paper is to study the feasibility of continuous and switch control modes for a brain-controlled wheelchair (BCW) using sensorimotor rhythms (SMR) modulated through a right-hand motor imagery task. Previous studies, which used a continuous navigation control with SMR, have reported the difficulty of maintaining the motor imagery task for a long time, especially for the forward command. The switch control has been presented as a proposal that may help to solve this issue since this task is only used temporary for either disabling or enabling the movement. Regarding the methodology, 10 of 15 able-bodied users, who had overcome the criterion of 30% error rate in the calibration phase, controlled the BCW using both paradigms. The navigation tasks consisted of a straight path divided in five sections: in three of them the users had to move forward, and in the other two the users had to maintain their position. To assess user performance in the device management, a usability approach was adopted, measuring the factors of effectiveness, efficiency, and satisfaction. Then, variables related to the time employed and commands selected by the user or parameters related to the confusion matrix were applied. In addition, the scores in NASA-TLX and two ad hoc questionnaires were considered to discuss the user experience controlling the wheelchair. Despite the results showed that the best system for a specific user relies on his/her abilities and preferences, the switch control mode obtained better accuracy (0.59 ± 0.17 for continuous and 0.72 ± 0.05 for switch). Furthermore, the switch paradigm can be recommended for the advance sections as with it users could complete the advance sections in less time (42.2 ± 28.7 s for continuous and 15.47 ± 3.43 s for switch), while the continuous mode seems to be better at keeping the wheelchair stopped (42.45 ± 16.01 s for continuous and 24.35 ± 10.94 s for switch).

Autonomy in Rehabilitation Robotics: An Intersection

Within the field of human rehabilitation, robotic machines are used both to rehabilitate the body and to perform functional tasks. Robotics autonomy able to perceive the external world and reason about high-level control decisions, however, seldom is present in these machines. For functional tasks in particular, autonomy could help to decrease the operational burden on the human and perhaps even to increase access-and this potential only grows as human motor impairments become more severe. There are however serious, and often subtle, considerations to introducing clinically-feasible robotics autonomy to rehabilitation robots and machines. Today the fields of robotics autonomy and rehabilitation robotics are largely separate. The topic of this article is at the intersection of these fields: the introduction of clinically-feasible autonomy solutions to rehabilitation robots, and opportunities for autonomy within the rehabilitation domain.

Scenario Screen: A Dynamic and Context Dependent P300 Stimulator Screen Aimed at Wheelchair Navigation Control

P300 spellers have been widely modified to implement nonspelling tasks. In this work, we propose a "scenario" stimulation screen that is a P300 speller variation to command a wheelchair. Our approach utilized a stimulation screen with an image background (scenario snapshot for a wheelchair) and stimulation markers arranged asymmetrically over relevant landmarks, such as suitable paths, doors, windows, and wall signs. Other scenario stimulation screen features were green/blue stimulation marker color scheme, variable Interstimulus Interval, single marker stimulus mode, and optimized stimulus sequence generator. Eighteen able-bodied subjects participated in the experiment; 78% had no experience in BCI usage. A waveform feature analysis and a Mann-Whitney U test performed over the pairs of target and nontarget coherent averages confirmed that 94% of the subjects elicit P300 (p < .005) on this modified stimulator. Least Absolute Shrinkage and Selection Operator optimization and Linear Discriminant Analysis were utilized for the automatic detection of P300. For evaluation with unseen data, target detection was computed (median sensitivity = 1.00 (0.78-1.00)), together with nontarget discrimination (median specificity = 1.00 (0.98-1.00)). The scenario screen adequately elicits P300 and seems suitable for commanding a wheelchair even when users have no previous experience on the BCI spelling task.

EEG-based emergency braking intention prediction for brain-controlled driving considering one electrode falling-off

This paper proposes a novel method of electroencephalography (EEG)-based driver emergency braking intention detection system for brain-controlled driving considering one electrode falling-off. First, whether one electrode falls off is discriminated based on EEG potentials. Then, the missing signals are estimated by using the signals collected from other channels based on multivariate linear regression. Finally, a linear decoder is applied to classify driver intentions. Experimental results show that the falling-off discrimination accuracy is 99.63% on average and the correlation coefficient and root mean squared error (RMSE) between the estimated and experimental data are 0.90 and 11.43 μV, respectively, on average. Given one electrode falls off, the system accuracy of the proposed intention prediction method is significantly higher than that of the original method (95.12% VS 79.11%) and is close to that (95.95%) of the original system under normal situations (i. e., no electrode falling-off).

A brain-controlled lower-limb exoskeleton for human gait training

Brain-computer interfaces have been a novel approach to translate human intentions into movement commands in robotic systems. This paper describes an electroencephalogram-based brain-controlled lower-limb exoskeleton for gait training, as a proof of concept towards rehabilitation with human-in-the-loop. Instead of using conventional single electroencephalography correlates, e.g., evoked P300 or spontaneous motor imagery, we propose a novel framework integrated two asynchronous signal modalities, i.e., sensorimotor rhythms (SMRs) and movement-related cortical potentials (MRCPs). We executed experiments in a biologically inspired and customized lower-limb exoskeleton where subjects (N = 6) actively controlled the robot using their brain signals. Each subject performed three consecutive sessions composed of offline training, online visual feedback testing, and online robot-control recordings. Post hoc evaluations were conducted including mental workload assessment, feature analysis, and statistics test. An average robot-control accuracy of 80.16% ± 5.44% was obtained with the SMR-based method, while estimation using the MRCP-based method yielded an average performance of 68.62% ± 8.55%. The experimental results showed the feasibility of the proposed framework with all subjects successfully controlled the exoskeleton. The current paradigm could be further extended to paraplegic patients in clinical trials.

An EOG-Based Human-Machine Interface for Wheelchair Control

OBJECTIVE

Nonmanual human-machine interfaces (HMIs) have been studied for wheelchair control with the aim of helping severely paralyzed individuals regain some mobility. The challenge is to rapidly, accurately, and sufficiently produce control commands, such as left and right turns, forward and backward motions, acceleration, deceleration, and stopping. In this paper, a novel electrooculogram (EOG) based HMI is proposed for wheelchair control.

METHODS

A total of 13 flashing buttons, each of which corresponds to a command, are presented in the graphical user interface. These buttons flash on a one-by-one manner in a predefined sequence. The user can select a button by blinking in sync with its flashes. The algorithm detects the eye blinks from a channel of vertical EOG data and determines the user's target button based on the synchronization between the detected blinks and the button's flashes.

RESULTS

For healthy subjects/patients with spinal cord injuries, the proposed HMI achieved an average accuracy of 96.7% / 91.7% and a response time of 3.53 s/3.67 s with 0 false positive rates (FPRs).

CONCLUSION

Using one channel of vertical EOG signals associated with eye blinks, the proposed HMI can accurately provide sufficient commands with a satisfactory response time.

SIGNIFICANCE

The proposed HMI provides a novel nonmanual approach for severely paralyzed individuals to control a wheelchair. Compared with a newly established EOG-based HMI, the proposed HMI can generate more commands with higher accuracy, lower FPR, and fewer electrodes.

Stop state classification in intracortical brain-machine-interface

Invasive brain-machine-interface (BMI) has the prospect to empower tetraplegic patients with independent mobility through the use of brain-controlled wheelchairs. For the practical and long-term use of such control systems, the system has to distinguish between stop and movement states and has to be robust to overcome non-stationarity in the brain signals. In this work, we investigates the non-stationarity of the stop state on neural data collected from a macaque trained to control a robotic platform to stop and move in left, right, forward directions We then propose a hybrid approach that employs both random forest and linear discriminant analysis (LDA). Using this approach, we performed offline decoding on 8 days of data collected over the course of three months during joystick control of the robotic platform. We compared the results of using the proposed approach with the use of LDA alone to perform direct classifications of stop, left, right and forward. The results showed an average performance increment of 22.7% using the proposed hybrid approach. The results yielded significant improvements during sessions where LDA showed a heavy bias towards the stop state. This suggests that the proposed hybrid approach addresses the non-stationarity in the stop state and subsequently facilitates a more accurate decoding of the movement states.

Progress in EEG-Based Brain Robot Interaction Systems

The most popular noninvasive Brain Robot Interaction (BRI) technology uses the electroencephalogram- (EEG-) based Brain Computer Interface (BCI), to serve as an additional communication channel, for robot control via brainwaves. This technology is promising for elderly or disabled patient assistance with daily life. The key issue of a BRI system is to identify human mental activities, by decoding brainwaves, acquired with an EEG device. Compared with other BCI applications, such as word speller, the development of these applications may be more challenging since control of robot systems via brainwaves must consider surrounding environment feedback in real-time, robot mechanical kinematics, and dynamics, as well as robot control architecture and behavior. This article reviews the major techniques needed for developing BRI systems. In this review article, we first briefly introduce the background and development of mind-controlled robot technologies. Second, we discuss the EEG-based brain signal models with respect to generating principles, evoking mechanisms, and experimental paradigms. Subsequently, we review in detail commonly used methods for decoding brain signals, namely, preprocessing, feature extraction, and feature classification, and summarize several typical application examples. Next, we describe a few BRI applications, including wheelchairs, manipulators, drones, and humanoid robots with respect to synchronous and asynchronous BCI-based techniques. Finally, we address some existing problems and challenges with future BRI techniques.

Navigation-synchronized multimodal control wheelchair from brain to alternative assistive technologies for persons with severe disabilities

Currently, electric wheelchairs are commonly used to improve mobility in disabled people. In severe cases, the user is unable to control the wheelchair by themselves because his/her motor functions are disabled. To restore mobility function, a brain-controlled wheelchair (BCW) would be a promising system that would allow the patient to control the wheelchair by their thoughts. P300 is a reliable brain electrical signal, a component of visual event-related potentials (ERPs), that could be used for interpreting user commands. This research aimed to propose a prototype BCW to allowed severe motor disabled patients to practically control a wheelchair for use in their home environment. The users were able to select from 9 possible destination commands in the automatic mode and from 4 directional commands (forward, backward, turn left and right) in the shared-control mode. These commands were selected via the designed P300 processing system. The wheelchair was steered to the desired location by the implemented navigation system. Safety of the user was ensured during wheelchair navigation due to the included obstacle detection and avoidance features. A combination of P300 and EOG was used as a hybrid BCW system. The user could fully operate the system such as enabling P300 detection system, mode shifting and stop/cancelation command by performing a different consecutive blinks to generate eye blinking patterns. The results revealed that the prototype BCW could be operated in either of the proposed modes. With the new design of the LED-based P300 stimulator, the average accuracies of the P300 detection algorithm in the shared-control and automatic modes were 95.31 and 83.42% with 3.09 and 3.79 bits/min, respectively. The P300 classification error was acceptable, as the user could cancel an incorrect command by blinking 2 times. Moreover, the proposed navigation system had a flexible design that could be interfaced with other assistive technologies. This research developed 3 alternative input modules: an eye tracker module and chin and hand controller modules. The user could select the most suitable assistive technology based on his/her level of disability. Other existing assistive technologies could also be connected to the proposed system in the future using the same protocol.

A BCI-based Environmental Control System for Patients with Severe Spinal Cord Injuries

This study proposes an event-related potential (ERP) BCI-based environmental control system that integrates household electrical appliances, a nursing bed, and an intelligent wheelchair to provide daily assistance to paralyzed patients with severe spinal cord injuries (SCIs).

METHODS

An asynchronous mode is used to switch the environmental control system on or off or to select a device (e.g., a TV) for achieving selfpaced control. In the asynchronous mode, we introduce several pseudo-keys and a verification mechanism to effectively reduce the false operation rate. By contrast, when the user selects a function of the device (e.g., a TV channel), a synchronous mode is used to improve the accuracy and speed of BCI detection. Two experiments involving six SCI patients were conducted separately in a nursing bed and a wheelchair, and the patients were instructed to control the nursing bed, the wheelchair, and household electrical appliances (an electric light, an air conditioner, and a TV).

RESULTS

The average false rate of BCI commands in the control state was 10.4%, whereas the average false operation ratio was 4.9% (a false BCI command might not necessarily result in a false operation according to our system design). During the idle state, there was an average of 0.97 false positives per minute, which did not result in any false operations.

CONCLUSION

All SCI patients could use the proposed ERP BCIbased environmental control system satisfactorily.

SIGNIFICANCE

The proposed ERP-based environmental control system could be used to assist patients with severe SCIs in their daily lives.

Static Versus Dynamic Decoding Algorithms in a Non-Invasive Body-Machine Interface

In this study, we consider a non-invasive body-machine interface that captures body motions still available to people with spinal cord injury (SCI) and maps them into a set of signals for controlling a computer user interface while engaging in a sustained level of mobility and exercise. We compare the effectiveness of two decoding algorithms that transform a high-dimensional body-signal vector into a lower dimensional control vector on six subjects with high-level SCI and eight controls. One algorithm is based on a static map from current body signals to the current value of the control vector set through principal component analysis (PCA), the other on dynamic mapping a segment of body signals to the value and the temporal derivatives of the control vector set through a Kalman filter. SCI and control participants performed straighter and smoother cursor movements with the Kalman algorithm during center-out reaching, but their movements were faster and more precise when using PCA. All participants were able to use the BMI's continuous, two-dimensional control to type on a virtual keyboard and play pong, and performance with both algorithms was comparable. However, seven of eight control participants preferred PCA as their method of virtual wheelchair control. The unsupervised PCA algorithm was easier to train and seemed sufficient to achieve a higher degree of learnability and perceived ease of use.

Independent Mobility Achieved through a Wireless Brain-Machine Interface

Individuals with tetraplegia lack independent mobility, making them highly dependent on others to move from one place to another. Here, we describe how two macaques were able to use a wireless integrated system to control a robotic platform, over which they were sitting, to achieve independent mobility using the neuronal activity in their motor cortices. The activity of populations of single neurons was recorded using multiple electrode arrays implanted in the arm region of primary motor cortex, and decoded to achieve brain control of the platform. We found that free-running brain control of the platform (which was not equipped with any machine intelligence) was fast and accurate, resembling the performance achieved using joystick control. The decoding algorithms can be trained in the absence of joystick movements, as would be required for use by tetraplegic individuals, demonstrating that the non-human primate model is a good pre-clinical model for developing such a cortically-controlled movement prosthetic. Interestingly, we found that the response properties of some neurons differed greatly depending on the mode of control (joystick or brain control), suggesting different roles for these neurons in encoding movement intention and movement execution. These results demonstrate that independent mobility can be achieved without first training on prescribed motor movements, opening the door for the implementation of this technology in persons with tetraplegia.

Review of real brain-controlled wheelchairs

This paper presents a review of the state of the art regarding wheelchairs driven by a brain-computer interface. Using a brain-controlled wheelchair (BCW), disabled users could handle a wheelchair through their brain activity, granting autonomy to move through an experimental environment. A classification is established, based on the characteristics of the BCW, such as the type of electroencephalographic signal used, the navigation system employed by the wheelchair, the task for the participants, or the metrics used to evaluate the performance. Furthermore, these factors are compared according to the type of signal used, in order to clarify the differences among them. Finally, the trend of current research in this field is discussed, as well as the challenges that should be solved in the future.

Brain-Computer Interface for Control of Wheelchair Using Fuzzy Neural Networks

The design of brain-computer interface for the wheelchair for physically disabled people is presented. The design of the proposed system is based on receiving, processing, and classification of the electroencephalographic (EEG) signals and then performing the control of the wheelchair. The number of experimental measurements of brain activity has been done using human control commands of the wheelchair. Based on the mental activity of the user and the control commands of the wheelchair, the design of classification system based on fuzzy neural networks (FNN) is considered. The design of FNN based algorithm is used for brain-actuated control. The training data is used to design the system and then test data is applied to measure the performance of the control system. The control of the wheelchair is performed under real conditions using direction and speed control commands of the wheelchair. The approach used in the paper allows reducing the probability of misclassification and improving the control accuracy of the wheelchair.

Queuing Network Modeling of Driver EEG Signals-Based Steering Control

Directly using brain signals rather than limbs to steer a vehicle may not only help disabled people to control an assistive vehicle, but also provide a complementary means of control for a wider driving community. In this paper, to simulate and predict driver performance in steering a vehicle with brain signals, we propose a driver brain-controlled steering model by combining an extended queuing network-based driver model with a brain-computer interface (BCI) performance model. Experimental results suggest that the proposed driver brain-controlled steering model has performance close to that of real drivers with good performance in brain-controlled driving. The brain-controlled steering model has potential values in helping develop a brain-controlled assistive vehicle. Furthermore, this study provides some insights into the simulation and prediction of the performance of using BCI systems to control other external devices (e.g., mobile robots).

A multi-command SSVEP-based BCI system based on single flickering frequency half-field steady-state visual stimulation

Steady-state visual evoked potentials (SSVEPs) are widely employed in brain-computer interface (BCI) applications, especially to control machines. However, the use of SSVEPs leads to eye fatigue and causes lower accuracy over the long term, particularly when multi-commands are required. Therefore, this paper proposes a half-field steady-state visual stimulation pattern and paradigm to increase the limited number of commands that can be achieved with existing SSVEP-based BCI methods. Following the theory of vision perception and existing half-field SSVEP-based BCI systems, the new stimulation pattern generates four commands using only one frequency flickering stimulus and has an average classification accuracy of approximately 75 %. According to the proposed stimulus pattern, using only one frequency without requiring users to stare directly at the flickering stimulus allows users to easily focus on the system and experience less visual fatigue compared to existing systems. Furthermore, new half-field SSVEP-based BCI systems are proposed, incorporating our proposed feature extraction and decision-making algorithm. Extracting the signal from the occipital area and using a reference electrode position at the parietal area yielded better results compared to the central area. In addition, we recommend using an LED or LCD as the visual stimulus device (at the recommended size), which yielded comparable results to our proposed feature extraction and decision-making algorithm. Finally, an application of the proposed system is demonstrated for real-time television control.

High-level hands-free control of wheelchair - a review

A vital perception in designing human-machine interfaces for people with disabilities is hands-free control. It is important in setting up communication among a machine and the part of human body. Nowadays, hands-free control is extensively used to help disabled people to control the wheelchair. The hands-free control functions are not only helpful to elderly and disabled, but also useful for normal people who might require using their hands for other tasks at the same time. As a result, normal people are capable of manipulating multiple tasks at the same time. The wheelchair is manoeuvred using brainwaves, eyes and facial muscles movements, voice, tongue, gesture. The mobility using a powered wheelchair improves the quality of life of the aged people and the people with mobility impairments significantly. These powered wheelchairs can be actuated using various high-level control interfaces. In this review, the use of natural biological signals and non-biological signal to do the hands-free control of powered wheelchair is presented in this article.

Time-Shift Correlation Algorithm for P300 Event Related Potential Brain-Computer Interface Implementation

A high efficient time-shift correlation algorithm was proposed to deal with the peak time uncertainty of P300 evoked potential for a P300-based brain-computer interface (BCI). The time-shift correlation series data were collected as the input nodes of an artificial neural network (ANN), and the classification of four LED visual stimuli was selected as the output node. Two operating modes, including fast-recognition mode (FM) and accuracy-recognition mode (AM), were realized. The proposed BCI system was implemented on an embedded system for commanding an adult-size humanoid robot to evaluate the performance from investigating the ground truth trajectories of the humanoid robot. When the humanoid robot walked in a spacious area, the FM was used to control the robot with a higher information transfer rate (ITR). When the robot walked in a crowded area, the AM was used for high accuracy of recognition to reduce the risk of collision. The experimental results showed that, in 100 trials, the accuracy rate of FM was 87.8% and the average ITR was 52.73 bits/min. In addition, the accuracy rate was improved to 92% for the AM, and the average ITR decreased to 31.27 bits/min. due to strict recognition constraints.

Commanding a Brain-Controlled Wheelchair Using Steady-State Somatosensory Evoked Potentials

In this work, we propose a novel brain-controlled wheelchair, one of the major applications of brain-machine interfaces (BMIs), that allows an individual with mobility impairments to perform daily living activities independently. Specifically, we propose to use a steady-state somatosensory evoked potential (SSSEP) paradigm, which elicits brain responses to tactile stimulation of specific frequencies, for a user's intention to control a wheelchair. In our system, a user had three possible commands by concentrating on one of three vibration stimuli, which were attached to the left-hand, right-hand, and right-foot, to selectively control the wheelchair. The three stimuli were associated with three wheelchair commands: turn-left, turn-right, and move-forward. From a machine learning perspective, we also devise a novel feature representation by combining spatial and spectral characteristics of brain signals. In order to validate the effectiveness of the proposed SSSEP-based system, we considered two different tasks: 1) a simple obstacle-avoidance task within a limited time and; 2) a driving task along the predefined trajectory of about 40 m length, where there were a narrow pathway, a door, and obstacles. In both experiments, we recruited 12 subjects and compared the average time of motor imagery (MI) and SSSEP-based controls to complete the task. With the SSSEP-based control, all subjects successfully completed the task without making any collision while four subjects failed it with MI-based control. It is also noteworthy that in terms of the average time to complete the task, the SSSEP-based control outperformed the MI-based control. In the other more challenging task, all subjects successfully reached the target location.

The Role of Audio-Visual Feedback in a Thought-Based Control of a Humanoid Robot: A BCI Study in Healthy and Spinal Cord Injured People

The efficient control of our body and successful interaction with the environment are possible through the integration of multisensory information. Brain-computer interface (BCI) may allow people with sensorimotor disorders to actively interact in the world. In this study, visual information was paired with auditory feedback to improve the BCI control of a humanoid surrogate. Healthy and spinal cord injured (SCI) people were asked to embody a humanoid robot and complete a pick-and-place task by means of a visual evoked potentials BCI system. Participants observed the remote environment from the robot's perspective through a head mounted display. Human-footsteps and computer-beep sounds were used as synchronous/asynchronous auditory feedback. Healthy participants achieved better placing accuracy when listening to human footstep sounds relative to a computer-generated sound. SCI people demonstrated more difficulty in steering the robot during asynchronous auditory feedback conditions. Importantly, subjective reports highlighted that the BCI mask overlaying the display did not limit the observation of the scenario and the feeling of being in control of the robot. Overall, the data seem to suggest that sensorimotor-related information may improve the control of external devices. Further studies are required to understand how the contribution of residual sensory channels could improve the reliability of BCI systems.

An intelligent wheelchair based on automated navigation and BCI techniques

In this paper, we propose an intelligent wheelchair system that relies on a brain computer interface (BCI) and automatic navigation. When in operation, candidate destinations and waypoints are automatically generated on the basis of the current environment. Then, the user selects a destination using a P300-based BCI. Finally, the navigation system plans a path and navigates the wheelchair to the determined destination. While the wheelchair is in motion, the user can issue a stop command with the BCI. Using our system, the mental burden of the user can be alleviated to a large degree. Furthermore, our system can adapt to changes in the environment. The experimental results demonstrated the effectiveness of our system.

A P300-Based Threshold-Free Brain Switch and Its Application in Wheelchair Control

The key issue of electroencephalography (EEG)-based brain switches is to detect the control and idle states in an asynchronous manner. Most existing methods rely on a threshold. However, it is often time consuming to select a satisfactory threshold, and the chosen threshold might be inappropriate over a long period of time due to the variability of the EEG signals. This paper presents a new P300-based threshold-free brain switch. Specifically, one target button and three pseudo buttons, which are intensified in a random order to produce P300 potential, are set in the graphical user interface. The user can issue a switch command by focusing on the target button. Two support vector machine (SVM) classifiers, namely, SVM1 and SVM2, are used in the detection algorithm. During detection, we first obtained four SVM scores, corresponding to the four flashing buttons, by applying SVM1 to the ongoing EEG. If the SVM score corresponding to the target button was negative or not at the maximum, then an idle state was determined. Moreover, if the target button had a maximum and positive score, then we fed the four SVM scores as features into SVM2 to further discriminate the control and idle states. As an application, this brain switch was used to produce a start/stop command for an intelligent wheelchair, of which the left, right, forward, backward functions were carried out by an autonomous navigation system. Several experiments were conducted with eight healthy subjects and five patients with spinal cord injuries (SCIs). The experimental results not only demonstrated the effectiveness of our approach but also illustrated the potential application for patients with SCIs.

APPLICATION OF S-TRANSFORM FOR AUTOMATED DETECTION OF VIGILANCE LEVEL USING EEG SIGNALS

This paper presents an S-transform-based Electroencephalogram channel optimization and feature extraction methodology for monitoring mental vigilance level of humans. Vigilance level detection methodology consists of four steps. In the first stage, two types of Electroencephalogram signals (alert and drowsy) are acquired from 30 healthy subjects and decomposed into sub-bands using the S-transform. In the second stage, permutation entropy of the S-transform coefficients is calculated and Electroencephalogram channel optimization is performed. S-transform-based statistical features are computed from the optimized Electroencephalogram channels, in the third stage. In the fourth stage, artificial intelligence techniques such as Least Square-Support Vector Machine, Artificial Neural Network and Naive Bayes Classifier are used for the classification of Electroencephalogram signals using extracted features. The performance of the feature extraction methodology is tested on the Electroencephalogram data of 30 healthy subjects. Experimental results ensured the effectiveness of proposed methodology for the estimation of mental vigilance level by using Electroencephalogram signals. It is observed that the Artificial Neural Network classifier is a good candidate for pre-emptive automatic vigilance level detection system for Brain-Computer Interface applications.

Endogenous Sensory Discrimination and Selection by a Fast Brain Switch for a High Transfer Rate Brain-Computer Interface

In this study, we present a novel multi-class brain-computer interface (BCI) for communication and control. In this system, the information processing is shared by the algorithm (computer) and the user (human). Specifically, an electro-tactile cycle was presented to the user, providing the choice (class) by delivering timely sensory input. The user discriminated these choices by his/her endogenous sensory ability and selected the desired choice with an intuitive motor task. This selection was detected by a fast brain switch based on real-time detection of movement-related cortical potentials from scalp EEG. We demonstrated the feasibility of such a system with a four-class BCI, yielding a true positive rate of  ∼ 80% and  ∼ 70%, and an information transfer rate of  ∼ 7 bits/min and  ∼ 5 bits/min, for the movement and imagination selection command, respectively. Furthermore, when the system was extended to eight classes, the throughput of the system was improved, demonstrating the capability of accommodating a large number of classes. Combining the endogenous sensory discrimination with the fast brain switch, the proposed system could be an effective, multi-class, gaze-independent BCI system for communication and control applications.

Towards personalized smart wheelchairs: Lessons learned from discovery interviews

We posit that it is necessary to investigate the personalization of smart wheelchairs in three aspects interfaces for interaction, controllers for action (top-level, middle-level, and low-level), and feedback in interaction. Our team has been selected as an Innovation Corps (I-Corps) Team by the National Science Foundation to pursue customer discovery research to explore the commercial viability of smart wheelchairs. Through the process, our team has performed more than 110 interviews with powered wheelchair users, manufacturers, therapists, policy makers, and non-profit organization staff. Our findings revealed that the acceptability of fully autonomous systems by the users is still challenging and highly-dependent on the severity of the disability. Furthermore, the cost, ease-of-use and personalization are the most important factors in commercializing smart wheelchair technologies.

Increasing N200 Potentials Via Visual Stimulus Depicting Humanoid Robot Behavior

Achieving recognizable visual event-related potentials plays an important role in improving the success rate in telepresence control of a humanoid robot via N200 or P300 potentials. The aim of this research is to intensively investigate ways to induce N200 potentials with obvious features by flashing robot images (images with meaningful information) and by flashing pictures containing only solid color squares (pictures with incomprehensible information). Comparative studies have shown that robot images evoke N200 potentials with recognizable negative peaks at approximately 260[Formula: see text]ms in the frontal and central areas. The negative peak amplitudes increase, on average, from [Formula: see text]V, induced by flashing the squares, to [Formula: see text]V, induced by flashing the robot images. The data analyses support that the N200 potentials induced by the robot image stimuli exhibit recognizable features. Compared with the square stimuli, the robot image stimuli increase the average accuracy rate by 9.92%, from 83.33% to 93.25%, and the average information transfer rate by 24.56[Formula: see text]bits/min, from 72.18[Formula: see text]bits/min to 96.74 bits/min, in a single repetition. This finding implies that the robot images might provide the subjects with more information to understand the visual stimuli meanings and help them more effectively concentrate on their mental activities.

Patient-specific cortical electrodes for sulcal and gyral implantation

PURPOSE

Noninvasive localization of certain brain functions may be mapped on a millimetre level. However, the interelectrode spacing of common clinical brain surface electrodes still remains around 10 mm. Here, we present details on development of electrodes for attaining higher quality electrocorticographic signals for use in functional brain mapping and brain-machine interface (BMI) technologies.

METHODS

We used platinum-plate-electrodes of 1-mm diameter to produce sheet electrodes after the creation of individualized molds using a 3-D printer and a press system that sandwiched the electrodes between personalized silicone sheets.

RESULTS

We created arrays to fit the surface curvature of the brain and inside the central sulcus, with interelectrode distances of 2.5 mm (a density of 16 times previous standard types). Rat experiments undertaken indicated no long term toxicity. We were also able to custom design, rapidly manufacture, safely implant, and confirm the efficacy of personalized electrodes, including the capability to attain meaningful high-gamma-band information in an amyotrophic lateral sclerosis patient.

CONCLUSION

We developed cortical sheet electrodes with a high-spatial resolution, tailor-made to match an individual's brain.

SIGNIFICANCE

This sheet electrode may contribute to the higher performance of BMI's.

Shared-control brain-computer interface for a two dimensional reaching task using EEG error-related potentials

One of the main problems of EEG-based brain computer interfaces (BCIs) is their low information rate, thus for complex tasks the user needs large amounts of time to solve the task. In an attempt to reduce this time and improve the application robustness, recent works have explored shared-control strategies where the device does not only execute the decoded commands, but it is also involved in executing the task. This work proposes a shared-control BCI using error potentials for a 2D reaching task with discrete actions and states. The proposed system has several interesting properties: the system is scalable without increasing the complexity of the user's mental task; the interaction is natural for the user, as the mental task is to monitor the device performance to promote its task learning (in this context the reaching task); and the system has the potential to be combined with additional brain signals to recover or learn from interaction errors. Online control experiments were performed with four subjects, showing that it was possible to reach a goal location from any starting point within a 5×5 grid in around 23 actions (about 19 seconds of EEG signal), both with fixed goals and goals freely chosen by the users.

A hybrid BCI for enhanced control of a telepresence robot

Motor-disabled end users have successfully driven a telepresence robot in a complex environment using a Brain-Computer Interface (BCI). However, to facilitate the interaction aspect that underpins the notion of telepresence, users must be able to voluntarily and reliably stop the robot at any moment, not just drive from point to point. In this work, we propose to exploit the user's residual muscular activity to provide a fast and reliable control channel, which can start/stop the telepresence robot at any moment. Our preliminary results show that not only does this hybrid approach increase the accuracy, but it also helps to reduce the workload and was the preferred control paradigm of all the participants.

Human-Machine Interface for the Control of Multi-Function Systems Based on Electrocutaneous Menu: Application to Multi-Grasp Prosthetic Hands

Modern assistive devices are very sophisticated systems with multiple degrees of freedom. However, an effective and user-friendly control of these systems is still an open problem since conventional human-machine interfaces (HMI) cannot easily accommodate the system’s complexity. In HMIs, the user is responsible for generating unique patterns of command signals directly triggering the device functions. This approach can be difficult to implement when there are many functions (necessitating many command patterns) and/or the user has a considerable impairment (limited number of available signal sources). In this study, we propose a novel concept for a general-purpose HMI where the controller and the user communicate bidirectionally to select the desired function. The system first presents possible choices to the user via electro-tactile stimulation; the user then acknowledges the desired choice by generating a single command signal. Therefore, the proposed approach simplifies the user communication interface (one signal to generate), decoding (one signal to recognize), and allows selecting from a number of options. To demonstrate the new concept the method was used in one particular application, namely, to implement the control of all the relevant functions in a state of the art commercial prosthetic hand without using any myoelectric channels. We performed experiments in healthy subjects and with one amputee to test the feasibility of the novel approach. The results showed that the performance of the novel HMI concept was comparable or, for some outcome measures, better than the classic myoelectric interfaces. The presented approach has a general applicability and the obtained results point out that it could be used to operate various assistive systems (e.g., prosthesis vs. wheelchair), or it could be integrated into other control schemes (e.g., myoelectric control, brain-machine interfaces) in order to improve the usability of existing low-bandwidth HMIs.

Upper Body-Based Power Wheelchair Control Interface for Individuals With Tetraplegia

Many power wheelchair control interfaces are not sufficient for individuals with severely limited upper limb mobility. The majority of controllers that do not rely on coordinated arm and hand movements provide users a limited vocabulary of commands and often do not take advantage of the user's residual motion. We developed a body-machine interface (BMI) that leverages the flexibility and customizability of redundant control by using high dimensional changes in shoulder kinematics to generate proportional control commands for a power wheelchair. In this study, three individuals with cervical spinal cord injuries were able to control a power wheelchair safely and accurately using only small shoulder movements. With the BMI, participants were able to achieve their desired trajectories and, after five sessions driving, were able to achieve smoothness that was similar to the smoothness with their current joystick. All participants were twice as slow using the BMI however improved with practice. Importantly, users were able to generalize training controlling a computer to driving a power wheelchair, and employed similar strategies when controlling both devices. Overall, this work suggests that the BMI can be an effective wheelchair control interface for individuals with high-level spinal cord injuries who have limited arm and hand control.

Control of a Wheelchair in an Indoor Environment Based on a Brain-Computer Interface and Automated Navigation

The concept of controlling a wheelchair using brain signals is promising. However, the continuous control of a wheelchair based on unstable and noisy electroencephalogram signals is unreliable and generates a significant mental burden for the user. A feasible solution is to integrate a brain-computer interface (BCI) with automated navigation techniques. This paper presents a brain-controlled intelligent wheelchair with the capability of automatic navigation. Using an autonomous navigation system, candidate destinations and waypoints are automatically generated based on the existing environment. The user selects a destination using a motor imagery (MI)-based or P300-based BCI. According to the determined destination, the navigation system plans a short and safe path and navigates the wheelchair to the destination. During the movement of the wheelchair, the user can issue a stop command with the BCI. Using our system, the mental burden of the user can be substantially alleviated. Furthermore, our system can adapt to changes in the environment. Two experiments based on MI and P300 were conducted to demonstrate the effectiveness of our system.

Usability and Workload of Access Technology for People With Severe Motor Impairment

Background. Eye trackers are widely used among people with amyotrophic lateral sclerosis, and their benefits to quality of life have been previously shown. On the contrary, Brain-computer interfaces (BCIs) are still quite a novel technology, which also serves as an access technology for people with severe motor impairment. Objective. To compare a visual P300-based BCI and an eye tracker in terms of information transfer rate (ITR), usability, and cognitive workload in users with motor impairments. Methods. Each participant performed 3 spelling tasks, over 4 total sessions, using an Internet browser, which was controlled by a spelling interface that was suitable for use with either the BCI or the eye tracker. At the end of each session, participants evaluated usability and cognitive workload of the system. Results. ITR and System Usability Scale (SUS) score were higher for the eye tracker (Wilcoxon signed-rank test: ITR T = 9, P = .016; SUS T = 12.50, P = .035). Cognitive workload was higher for the BCI ( T = 4; P = .003). Conclusions. Although BCIs could be potentially useful for people with severe physical disabilities, we showed that the usability of BCIs based on the visual P300 remains inferior to eye tracking. We suggest that future research on visual BCIs should use eye tracking–based control as a comparison to evaluate performance or focus on nonvisual paradigms for persons who have lost gaze control.