1
|
Fernández-Rodríguez Á, Martínez-Cagigal V, Santamaría-Vázquez E, Ron-Angevin R, Hornero R. Influence of spatial frequency in visual stimuli for cVEP-based BCIs: evaluation of performance and user experience. Front Hum Neurosci 2023; 17:1288438. [PMID: 38021231 PMCID: PMC10667696 DOI: 10.3389/fnhum.2023.1288438] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/04/2023] [Accepted: 10/23/2023] [Indexed: 12/01/2023] Open
Abstract
Code-modulated visual evoked potentials (c-VEPs) are an innovative control signal utilized in brain-computer interfaces (BCIs) with promising performance. Prior studies on steady-state visual evoked potentials (SSVEPs) have indicated that the spatial frequency of checkerboard-like stimuli influences both performance and user experience. Spatial frequency refers to the dimensions of the individual squares comprising the visual stimulus, quantified in cycles (i.e., number of black-white squares pairs) per degree of visual angle. However, the specific effects of this parameter on c-VEP-based BCIs remain unexplored. Therefore, the objective of this study is to investigate the role of spatial frequency of checkerboard-like visual stimuli in a c-VEP-based BCI. Sixteen participants evaluated selection matrices with eight spatial frequencies: C001 (0 c/°, 1×1 squares), C002 (0.15 c/°, 2×2 squares), C004 (0.3 c/°, 4×4 squares), C008 (0.6 c/°, 8×8 squares), C016 (1.2 c/°, 16×16 squares), C032 (2.4 c/°, 32×32 squares), C064 (4.79 c/°, 64×64 squares), and C128 (9.58 c/°, 128×128 squares). These conditions were tested in an online spelling task, which consisted of 18 trials each conducted on a 3×3 command interface. In addition to accuracy and information transfer rate (ITR), subjective measures regarding comfort, ocular irritation, and satisfaction were collected. Significant differences in performance and comfort were observed based on different stimulus spatial frequencies. Although all conditions achieved mean accuracy over 95% after 2.1 s of trial duration, C016 stood out in terms user experience. The proposed condition not only achieved a mean accuracy of 96.53% and 164.54 bits/min with a trial duration of 1.05s, but also was reported to be significantly more comfortable than the traditional C001 stimulus. Since both features are key for BCI development, higher spatial frequencies than the classical black-to-white stimulus might be more adequate for c-VEP systems. Hence, we assert that the spatial frequency should be carefully considered in the development of future applications for c-VEP-based BCIs.
Collapse
Affiliation(s)
| | - Víctor Martínez-Cagigal
- Grupo de Ingeniería Biomédica, Universidad de Valladolid, Valladolid, Spain
- Centro de Investigación Biomédica en Red de Bioingeniería, Biomateriales y Nanomedicina, Valladolid, Spain
| | - Eduardo Santamaría-Vázquez
- Grupo de Ingeniería Biomédica, Universidad de Valladolid, Valladolid, Spain
- Centro de Investigación Biomédica en Red de Bioingeniería, Biomateriales y Nanomedicina, Valladolid, Spain
| | - Ricardo Ron-Angevin
- UMA-BCI Group, Departamento de Tecnología Electrónica, Universidad de Málaga, Malaga, Spain
| | - Roberto Hornero
- Grupo de Ingeniería Biomédica, Universidad de Valladolid, Valladolid, Spain
- Centro de Investigación Biomédica en Red de Bioingeniería, Biomateriales y Nanomedicina, Valladolid, Spain
| |
Collapse
|
2
|
Xu B, Liu D, Xue M, Miao M, Hu C, Song A. Continuous shared control of a mobile robot with brain-computer interface and autonomous navigation for daily assistance. Comput Struct Biotechnol J 2023; 22:3-16. [PMID: 37600142 PMCID: PMC10433001 DOI: 10.1016/j.csbj.2023.07.033] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2023] [Revised: 07/04/2023] [Accepted: 07/22/2023] [Indexed: 08/22/2023] Open
Abstract
Although the electroencephalography (EEG) based brain-computer interface (BCI) has been successfully developed for rehabilitation and assistance, it is still challenging to achieve continuous control of a brain-actuated mobile robot system. In this study, we propose a continuous shared control strategy combining continuous BCI and autonomous navigation for a mobile robot system. The weight of shared control is designed to dynamically adjust the fusion of continuous BCI control and autonomous navigation. During this process, the system uses the visual-based simultaneous localization and mapping (SLAM) method to construct environmental maps. After obtaining the global optimal path, the system utilizes the brain-based shared control dynamic window approach (BSC-DWA) to evaluate safe and reachable trajectories while considering shared control velocity. Eight subjects participated in two-stage training, and six of these eight subjects participated in online shared control experiments. The training results demonstrated that naïve subjects could achieve continuous control performance with an average percent valid correct rate of approximately 97 % and an average total correct rate of over 80 %. The results of online shared control experiments showed that all of the subjects could complete navigation tasks in an unknown corridor with continuous shared control. Therefore, our experiments verified the feasibility and effectiveness of the proposed system combining continuous BCI, shared control, autonomous navigation, and visual SLAM. The proposed continuous shared control framework shows great promise in BCI-driven tasks, especially navigation tasks for brain-driven assistive mobile robots and wheelchairs in daily applications.
Collapse
Affiliation(s)
- Baoguo Xu
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Deping Liu
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Muhui Xue
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| | - Minmin Miao
- School of Information Engineering, Huzhou University, Huzhou 313000, China
| | - Cong Hu
- Guangxi Key Laboratory of Automatic Detecting Technology and Instruments, Guilin University of Electronic Technology, Guilin 541004, China
| | - Aiguo Song
- State Key Laboratory of Bioelectronics, Jiangsu Key Laboratory of Remote Measurement and Control, School of Instrument Science and Engineering, Southeast University, Nanjing 210096, China
| |
Collapse
|
3
|
Jiang L, Luo C, Liao Z, Li X, Chen Q, Jin Y, Lu K, Zhang D. SmartRolling: A human–machine interface for wheelchair control using EEG and smart sensing techniques. Inf Process Manag 2023. [DOI: 10.1016/j.ipm.2022.103262] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
|
4
|
Li H, Liu M, Yu X, Zhu J, Wang C, Chen X, Feng C, Leng J, Zhang Y, Xu F. Coherence based graph convolution network for motor imagery-induced EEG after spinal cord injury. Front Neurosci 2023; 16:1097660. [PMID: 36711141 PMCID: PMC9880407 DOI: 10.3389/fnins.2022.1097660] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2022] [Accepted: 12/28/2022] [Indexed: 01/15/2023] Open
Abstract
Background Spinal cord injury (SCI) may lead to impaired motor function, autonomic nervous system dysfunction, and other dysfunctions. Brain-computer Interface (BCI) system based on motor imagery (MI) can provide more scientific and effective treatment solutions for SCI patients. Methods According to the interaction between brain regions, a coherence-based graph convolutional network (C-GCN) method is proposed to extract the temporal-frequency-spatial features and functional connectivity information of EEG signals. The proposed algorithm constructs multi-channel EEG features based on coherence networks as graphical signals and then classifies MI tasks. Different from the traditional graphical convolutional neural network (GCN), the C-GCN method uses the coherence network of EEG signals to determine MI-related functional connections, which are used to represent the intrinsic connections between EEG channels in different rhythms and different MI tasks. EEG data of SCI patients and healthy subjects have been analyzed, where healthy subjects served as the control group. Results The experimental results show that the C-GCN method can achieve the best classification performance with certain reliability and stability, the highest classification accuracy is 96.85%. Conclusion The proposed framework can provide an effective theoretical basis for the rehabilitation treatment of SCI patients.
Collapse
Affiliation(s)
- Han Li
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China
| | - Ming Liu
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China
| | - Xin Yu
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China
| | - JianQun Zhu
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China
| | - Chongfeng Wang
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China
| | - Xinyi Chen
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China
| | - Chao Feng
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China,*Correspondence: Chao Feng,
| | - Jiancai Leng
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China,Jiancai Leng,
| | - Yang Zhang
- Rehabilitation Center, Qilu Hospital of Shandong University, Jinan, China,Yang Zhang,
| | - Fangzhou Xu
- International School for Optoelectronic Engineering, Qilu University of Technology, Shandong Academy of Sciences, Jinan, China,Fangzhou Xu,
| |
Collapse
|
5
|
Zhao SN, Cui Y, He Y, He Z, Diao Z, Peng F, Cheng C. Teleoperation control of a wheeled mobile robot based on Brain-machine Interface. MATHEMATICAL BIOSCIENCES AND ENGINEERING : MBE 2023; 20:3638-3660. [PMID: 36899597 DOI: 10.3934/mbe.2023170] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/18/2023]
Abstract
This paper presents a novel teleoperation system using Electroencephalogram (EEG) to control the motion of a wheeled mobile robot (WMR). Different from the other traditional motion controlling method, the WMR is braked with the EEG classification results. Furthermore, the EEG will be induced by using the online BMI (Brain Machine Interface) system, and adopting the non-intrusion induced mode SSVEP (steady state visually evoked potentials). Then, user's motion intention can be recognized by canonical correlation analysis (CCA) classifier, which will be converted into motion commands of the WMR. Finally, the teleoperation technique is utilized to manage the information of the movement scene and adjust the control instructions based on the real-time information. Bezier curve is used to parameterize the path planning of the robot, and the trajectory can be adjusted in real time by EEG recognition results. A motion controller based on error model is proposed to track the planned trajectory by using velocity feedback control, providing excellent track tracking performance. Finally, the feasibility and performance of the proposed teleoperation brain-controlled WMR system are verified using demonstration experiments.
Collapse
Affiliation(s)
- Su-Na Zhao
- College of Electrical and Information Engineering, Zhengzhou University of Light Industry, Zhengzhou 450000, China
| | - Yingxue Cui
- College of Electrical and Information Engineering, Zhengzhou University of Light Industry, Zhengzhou 450000, China
| | - Yan He
- College of Electrical and Information Engineering, Zhengzhou University of Light Industry, Zhengzhou 450000, China
| | - Zhendong He
- College of Electrical and Information Engineering, Zhengzhou University of Light Industry, Zhengzhou 450000, China
| | - Zhihua Diao
- College of Electrical and Information Engineering, Zhengzhou University of Light Industry, Zhengzhou 450000, China
| | - Fang Peng
- Zhongshan Institute, University of Electronic Science and Technology of China, Zhongshan 528402, China
| | - Chao Cheng
- Key Laboratory of Bionic Engineering, Ministry of Education, Jilin University, Changchun 130022, China
- Weihai Institute for Bionics, Jilin University, Weihai 264402, China
| |
Collapse
|
6
|
Wang X, Chen HT, Lin CT. Error-related potential-based shared autonomy via deep recurrent reinforcement learning. J Neural Eng 2022; 19. [PMID: 36541532 DOI: 10.1088/1741-2552/aca4fb] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2022] [Accepted: 11/22/2022] [Indexed: 11/23/2022]
Abstract
Objective.Error-related potential (ErrP)-based brain-computer interfaces (BCIs) have received a considerable amount of attention in the human-robot interaction community. In contrast to traditional BCI, which requires continuous and explicit commands from an operator, ErrP-based BCI leverages the ErrP, which is evoked when an operator observes unexpected behaviours from the robot counterpart. This paper proposes a novel shared autonomy model for ErrP-based human-robot interaction.Approach.We incorporate ErrP information provided by a BCI as useful observations for an agent and formulate the shared autonomy problem as a partially observable Markov decision process. A recurrent neural network-based actor-critic model is used to address the uncertainty in the ErrP signal. We evaluate the proposed framework in a simulated human-in-the-loop robot navigation task with both simulated users and real users.Main results.The results show that the proposed ErrP-based shared autonomy model enables an autonomous robot to complete navigation tasks more efficiently. In a simulation with 70% ErrP accuracy, agents completed the task 14.1% faster than in the no ErrP condition, while with real users, agents completed the navigation task 14.9% faster.Significance.The evaluation results confirmed that the shared autonomy via deep recurrent reinforcement learning is an effective way to deal with uncertain human feedback in a complex human-robot interaction task.
Collapse
Affiliation(s)
- Xiaofei Wang
- Computational Intelligence and Brain Computer Interface Lab, School of Computer Science, University of Technology, Sydney, NSW 2007, Australia
| | - Hsiang-Ting Chen
- School of Computer Science, University of Adelaide, Adelaide, SA 5005, Australia
| | - Chin-Teng Lin
- Computational Intelligence and Brain Computer Interface Lab, School of Computer Science, University of Technology, Sydney, NSW 2007, Australia
| |
Collapse
|
7
|
Tonin L, Perdikis S, Kuzu TD, Pardo J, Orset B, Lee K, Aach M, Schildhauer TA, Martínez-Olivera R, Millán JDR. Learning to control a BMI-driven wheelchair for people with severe tetraplegia. iScience 2022; 25:105418. [PMID: 36590466 PMCID: PMC9801246 DOI: 10.1016/j.isci.2022.105418] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/25/2022] [Revised: 08/14/2022] [Accepted: 10/18/2022] [Indexed: 11/19/2022] Open
Abstract
Mind-controlled wheelchairs are an intriguing assistive mobility solution applicable in complete paralysis. Despite progress in brain-machine interface (BMI) technology, its translation remains elusive. The primary objective of this study is to probe the hypothesis that BMI skill acquisition by end-users is fundamental to control a non-invasive brain-actuated intelligent wheelchair in real-world settings. We demonstrate that three tetraplegic spinal-cord injury users could be trained to operate a non-invasive, self-paced thought-controlled wheelchair and execute complex navigation tasks. However, only the two users exhibiting increasing decoding performance and feature discriminancy, significant neuroplasticity changes and improved BMI command latency, achieved high navigation performance. In addition, we show that dexterous, continuous control of robots is possible through low-degree of freedom, discrete and uncertain control channels like a motor imagery BMI, by blending human and artificial intelligence through shared-control methodologies. We posit that subject learning and shared-control are the key components paving the way for translational non-invasive BMI.
Collapse
Affiliation(s)
- Luca Tonin
- Department of Information Engineering, University of Padova, Padova, Italy,Padova Neuroscience Center, University of Padova, Padova, Italy
| | - Serafeim Perdikis
- Brain-Computer Interfaces and Neural Engineering Laboratory, School of Computer Science and Electronic Engineering, University of Essex, Colchester, UK
| | - Taylan Deniz Kuzu
- Klinik für Neurochirurgie und Wirbelsäulenchirurgie, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - Jorge Pardo
- Klinik für Neurochirurgie und Wirbelsäulenchirurgie, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - Bastien Orset
- École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Kyuhwa Lee
- Wyss Center for Bio and Neuroengineering, Geneva, Switzerland
| | - Mirko Aach
- Chirurgische Universitätsklinik und Poliklinik, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - Thomas Armin Schildhauer
- Chirurgische Universitätsklinik und Poliklinik, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - Ramón Martínez-Olivera
- Klinik für Neurochirurgie und Wirbelsäulenchirurgie, Universitätsklinikum Bergmannsheil Bochum, Ruhr-Universität Bochum, Bochum, Germany
| | - José del R. Millán
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, USA,Department of Neurology, The University of Texas at Austin, Austin, TX, USA,Corresponding author
| |
Collapse
|
8
|
Brain-computer interface (BCI)-generated speech to control domotic devices. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2022.08.068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|
9
|
Chen W, Chen SK, Liu YH, Chen YJ, Chen CS. An Electric Wheelchair Manipulating System Using SSVEP-Based BCI System. BIOSENSORS 2022; 12:bios12100772. [PMID: 36290910 PMCID: PMC9599534 DOI: 10.3390/bios12100772] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/07/2022] [Revised: 09/05/2022] [Accepted: 09/16/2022] [Indexed: 11/22/2022]
Abstract
Most people with motor disabilities use a joystick to control an electric wheelchair. However, those who suffer from multiple sclerosis or amyotrophic lateral sclerosis may require other methods to control an electric wheelchair. This study implements an electroencephalography (EEG)-based brain–computer interface (BCI) system and a steady-state visual evoked potential (SSVEP) to manipulate an electric wheelchair. While operating the human–machine interface, three types of SSVEP scenarios involving a real-time virtual stimulus are displayed on a monitor or mixed reality (MR) goggles to produce the EEG signals. Canonical correlation analysis (CCA) is used to classify the EEG signals into the corresponding class of command and the information transfer rate (ITR) is used to determine the effect. The experimental results show that the proposed SSVEP stimulus generates the EEG signals because of the high classification accuracy of CCA. This is used to control an electric wheelchair along a specific path. Simultaneous localization and mapping (SLAM) is the mapping method that is available in the robotic operating software (ROS) platform that is used for the wheelchair system for this study.
Collapse
Affiliation(s)
- Wen Chen
- Graduate Institute of Automation Technology, National Taipei University of Technology, Taipei 10608, Taiwan
| | - Shih-Kang Chen
- Department of Mechatronics Control, Industrial Technology Research Institute, Hsinchu 310401, Taiwan
| | - Yi-Hung Liu
- Department of Mechanical Engineering, National Taiwan University of Science and Technology, Taipei 106335, Taiwan
| | - Yu-Jen Chen
- Department of Radiation Oncology, MacKay Memorial Hospital, Taipei 10449, Taiwan
| | - Chin-Sheng Chen
- Graduate Institute of Automation Technology, National Taipei University of Technology, Taipei 10608, Taiwan
- Correspondence: ; Tel.: +886-2-27712171 (ext. 4325)
| |
Collapse
|
10
|
Liu K, Yu Y, Liu Y, Tang J, Liang X, Chu X, Zhou Z. A novel brain-controlled wheelchair combined with computer vision and augmented reality. Biomed Eng Online 2022; 21:50. [PMID: 35883092 PMCID: PMC9327337 DOI: 10.1186/s12938-022-01020-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2022] [Accepted: 07/11/2022] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Brain-controlled wheelchairs (BCWs) are important applications of brain-computer interfaces (BCIs). Currently, most BCWs are semiautomatic. When users want to reach a target of interest in their immediate environment, this semiautomatic interaction strategy is slow. METHODS To this end, we combined computer vision (CV) and augmented reality (AR) with a BCW and proposed the CVAR-BCW: a BCW with a novel automatic interaction strategy. The proposed CVAR-BCW uses a translucent head-mounted display (HMD) as the user interface, uses CV to automatically detect environments, and shows the detected targets through AR technology. Once a user has chosen a target, the CVAR-BCW can automatically navigate to it. For a few scenarios, the semiautomatic strategy might be useful. We integrated a semiautomatic interaction framework into the CVAR-BCW. The user can switch between the automatic and semiautomatic strategies. RESULTS We recruited 20 non-disabled subjects for this study and used the accuracy, information transfer rate (ITR), and average time required for the CVAR-BCW to reach each designated target as performance metrics. The experimental results showed that our CVAR-BCW performed well in indoor environments: the average accuracies across all subjects were 83.6% (automatic) and 84.1% (semiautomatic), the average ITRs were 8.2 bits/min (automatic) and 8.3 bits/min (semiautomatic), the average times required to reach a target were 42.4 s (automatic) and 93.4 s (semiautomatic), and the average workloads and degrees of fatigue for the two strategies were both approximately 20. CONCLUSIONS Our CVAR-BCW provides a user-centric interaction approach and a good framework for integrating more advanced artificial intelligence technologies, which may be useful in the field of disability assistance.
Collapse
Affiliation(s)
- Kaixuan Liu
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha, 410073, Hunan, China
| | - Yang Yu
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha, 410073, Hunan, China.
| | - Yadong Liu
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha, 410073, Hunan, China
| | - Jingsheng Tang
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha, 410073, Hunan, China
| | - Xinbin Liang
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha, 410073, Hunan, China
| | - Xingxing Chu
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha, 410073, Hunan, China
| | - Zongtan Zhou
- College of Intelligence Science and Technology, National University of Defense Technology, Changsha, 410073, Hunan, China
| |
Collapse
|
11
|
Cross-Platform Implementation of an SSVEP-Based BCI for the Control of a 6-DOF Robotic Arm. SENSORS 2022; 22:s22135000. [PMID: 35808498 PMCID: PMC9269816 DOI: 10.3390/s22135000] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/02/2022] [Revised: 06/28/2022] [Accepted: 06/29/2022] [Indexed: 11/16/2022]
Abstract
Robotics has been successfully applied in the design of collaborative robots for assistance to people with motor disabilities. However, man-machine interaction is difficult for those who suffer severe motor disabilities. The aim of this study was to test the feasibility of a low-cost robotic arm control system with an EEG-based brain-computer interface (BCI). The BCI system relays on the Steady State Visually Evoked Potentials (SSVEP) paradigm. A cross-platform application was obtained in C++. This C++ platform, together with the open-source software Openvibe was used to control a Stäubli robot arm model TX60. Communication between Openvibe and the robot was carried out through the Virtual Reality Peripheral Network (VRPN) protocol. EEG signals were acquired with the 8-channel Enobio amplifier from Neuroelectrics. For the processing of the EEG signals, Common Spatial Pattern (CSP) filters and a Linear Discriminant Analysis classifier (LDA) were used. Five healthy subjects tried the BCI. This work allowed the communication and integration of a well-known BCI development platform such as Openvibe with the specific control software of a robot arm such as Stäubli TX60 using the VRPN protocol. It can be concluded from this study that it is possible to control the robotic arm with an SSVEP-based BCI with a reduced number of dry electrodes to facilitate the use of the system.
Collapse
|
12
|
Sort and Deep-SORT Based Multi-Object Tracking for Mobile Robotics: Evaluation with New Data Association Metrics. APPLIED SCIENCES-BASEL 2022. [DOI: 10.3390/app12031319] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 02/01/2023]
Abstract
Multi-Object Tracking (MOT) techniques have been under continuous research and increasingly applied in a diverse range of tasks. One area in particular concerns its application in navigation tasks of assistive mobile robots, with the aim to increase the mobility and autonomy of people suffering from mobility decay, or severe motor impairments, due to muscular, neurological, or osteoarticular decay. Therefore, in this work, having in view navigation tasks for assistive mobile robots, an evaluation study of two MOTs by detection algorithms, SORT and Deep-SORT, is presented. To improve the data association of both methods, which are solved as a linear assignment problem with a generated cost matrix, a set of new object tracking data association cost matrices based on intersection over union, Euclidean distances, and bounding box metrics is proposed. For the evaluation of the MOT by detection in a real-time pipeline, the YOLOv3 is used to detect and classify the objects available on images. In addition, to perform the proposed evaluation aiming at assistive platforms, the ISR Tracking dataset, which represents the object conditions under which real robotic platforms may navigate, is presented. Experimental evaluations were also carried out on the MOT17 dataset. Promising results were achieved by the proposed object tracking data association cost matrices, showing an improvement in the majority of the MOT evaluation metrics compared to the default data association cost matrix. In addition, promising frame rate values were attained by the pipeline composed of the detector and the tracking module.
Collapse
|
13
|
Traded and combined cooperative control of a smart wheelchair. ROBOTICA 2022. [DOI: 10.1017/s0263574721001855] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Abstract
This article deals with a human–machine cooperative system for the control of a smart wheelchair for people with motor disabilities. The choice of a traded control mode is first argued. The paper then pursues two objectives. The first is to describe the design of the cooperative system by focusing on the dialogue and the interaction between the pilot and the robot. The second objective is to introduce a new cooperative mode. In this one, three features are proposed: two semi-autonomous features, a wall following and a doorway crossing, during which the user can intervene punctually to rectify a trajectory or a path, and an assisted mode where, conversely, the machine intervenes in a manual control to avoid obstacles. This mode of intervention of an entity, human or machine, supervising a movement controlled by the other is referred as “combined control.” Examples of scenarios exploiting the cooperative capabilities of the system are presented and discussed.
Collapse
|
14
|
Oladele DA, Markus ED, Abu-Mahfouz AM. Adaptability of Assistive Mobility Devices and the Role of the Internet of Medical Things: Comprehensive Review. JMIR Rehabil Assist Technol 2021; 8:e29610. [PMID: 34779786 PMCID: PMC8663709 DOI: 10.2196/29610] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/13/2021] [Revised: 06/29/2021] [Accepted: 09/12/2021] [Indexed: 01/22/2023] Open
Abstract
Background With the projected upsurge in the percentage of people with some form of disability, there has been a significant increase in the need for assistive mobility devices. However, for mobility aids to be effective, such devices should be adapted to the user’s needs. This can be achieved by improving the confidence of the acquired information (interaction between the user, the environment, and the device) following design specifications. Therefore, there is a need for literature review on the adaptability of assistive mobility devices. Objective In this study, we aim to review the adaptability of assistive mobility devices and the role of the internet of medical things in terms of the acquired information for assistive mobility devices. We review internet-enabled assistive mobility technologies and non–internet of things (IoT) assistive mobility devices. These technologies will provide awareness of the status of adaptive mobility technology and serve as a source and reference regarding information to health care professionals and researchers. Methods We performed a literature review search on the following databases of academic references and journals: Google Scholar, ScienceDirect, Institute of Electrical and Electronics Engineers, Springer, and websites of assistive mobility and foundations presenting studies on assistive mobility found through a generic Google search (including the World Health Organization website). The following keywords were used: assistive mobility OR assistive robots, assistive mobility devices, internet-enabled assistive mobility technologies, IoT Framework OR IoT Architecture AND for Healthcare, assisted navigation OR autonomous navigation, mobility AND aids OR devices, adaptability of assistive technology, adaptive mobility devices, pattern recognition, autonomous navigational systems, human-robot interfaces, motor rehabilitation devices, perception, and ambient assisted living. Results We identified 13,286 results (excluding titles that were not relevant to this study). Then, through a narrative review, we selected 189 potential studies (189/13,286, 1.42%) from the existing literature on the adaptability of assistive mobility devices and IoT frameworks for assistive mobility and conducted a critical analysis. Of the 189 potential studies, 82 (43.4%) were selected for analysis after meeting the inclusion criteria. On the basis of the type of technologies presented in the reviewed articles, we proposed a categorization of the adaptability of smart assistive mobility devices in terms of their interaction with the user (user system interface), perception techniques, and communication and sensing frameworks. Conclusions We discussed notable limitations of the reviewed literature studies. The findings revealed that an improvement in the adaptation of assistive mobility systems would require a reduction in training time and avoidance of cognitive overload. Furthermore, sensor fusion and classification accuracy are critical for achieving real-world testing requirements. Finally, the trade-off between cost and performance should be considered in the commercialization of these devices.
Collapse
Affiliation(s)
- Daniel Ayo Oladele
- Department of Electrical, Electronic and Computer Engineering, Central University of Technology, Bloemfontein, South Africa
| | - Elisha Didam Markus
- Department of Electrical, Electronic and Computer Engineering, Central University of Technology, Bloemfontein, South Africa
| | | |
Collapse
|
15
|
Megalingam RK, Rajendraprasad A, Raj A, Raghavan D, Teja CR, Sreekanth S, Sankaran R. Self-E: a self-driving wheelchair for elders and physically challenged. INTERNATIONAL JOURNAL OF INTELLIGENT ROBOTICS AND APPLICATIONS 2021. [DOI: 10.1007/s41315-021-00209-9] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022]
|
16
|
Palumbo A, Gramigna V, Calabrese B, Ielpo N. Motor-Imagery EEG-Based BCIs in Wheelchair Movement and Control: A Systematic Literature Review. SENSORS (BASEL, SWITZERLAND) 2021; 21:6285. [PMID: 34577493 PMCID: PMC8473300 DOI: 10.3390/s21186285] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/07/2021] [Revised: 09/09/2021] [Accepted: 09/14/2021] [Indexed: 02/07/2023]
Abstract
The pandemic emergency of the coronavirus disease 2019 (COVID-19) shed light on the need for innovative aids, devices, and assistive technologies to enable people with severe disabilities to live their daily lives. EEG-based Brain-Computer Interfaces (BCIs) can lead individuals with significant health challenges to improve their independence, facilitate participation in activities, thus enhancing overall well-being and preventing impairments. This systematic review provides state-of-the-art applications of EEG-based BCIs, particularly those using motor-imagery (MI) data, to wheelchair control and movement. It presents a thorough examination of the different studies conducted since 2010, focusing on the algorithm analysis, features extraction, features selection, and classification techniques used as well as on wheelchair components and performance evaluation. The results provided in this paper could highlight the limitations of current biomedical instrumentations applied to people with severe disabilities and bring focus to innovative research topics.
Collapse
Affiliation(s)
- Arrigo Palumbo
- Department of Medical and Surgical Sciences, “Magna Græcia” University, 88100 Catanzaro, Italy; (A.P.); (B.C.); (N.I.)
| | - Vera Gramigna
- Neuroscience Research Center, Magna Græcia University, 88100 Catanzaro, Italy
| | - Barbara Calabrese
- Department of Medical and Surgical Sciences, “Magna Græcia” University, 88100 Catanzaro, Italy; (A.P.); (B.C.); (N.I.)
| | - Nicola Ielpo
- Department of Medical and Surgical Sciences, “Magna Græcia” University, 88100 Catanzaro, Italy; (A.P.); (B.C.); (N.I.)
| |
Collapse
|
17
|
A Bibliometric Analysis of Human-Machine Interaction Methodology for Electric-Powered Wheelchairs Driving from 1998 to 2020. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2021; 18:ijerph18147567. [PMID: 34300017 PMCID: PMC8304937 DOI: 10.3390/ijerph18147567] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 03/23/2021] [Revised: 05/17/2021] [Accepted: 07/10/2021] [Indexed: 11/17/2022]
Abstract
Electric power wheelchairs (EPWs) enhance the mobility capability of the elderly and the disabled, while the human-machine interaction (HMI) determines how well the human intention will be precisely delivered and how human-machine system cooperation will be efficiently conducted. A bibliometric quantitative analysis of 1154 publications related to this research field, published between 1998 and 2020, was conducted. We identified the development status, contributors, hot topics, and potential future research directions of this field. We believe that the combination of intelligence and humanization of an EPW HMI system based on human-machine collaboration is an emerging trend in EPW HMI methodology research. Particular attention should be paid to evaluating the applicability and benefits of the EPW HMI methodology for the users, as well as how much it contributes to society. This study offers researchers a comprehensive understanding of EPW HMI studies in the past 22 years and latest trends from the evolutionary footprints and forward-thinking insights regarding future research.
Collapse
|
18
|
Udupa S, Kamat VR, Menassa CC. Shared autonomy in assistive mobile robots: a review. Disabil Rehabil Assist Technol 2021:1-22. [PMID: 34133906 DOI: 10.1080/17483107.2021.1928778] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/21/2022]
Abstract
PURPOSE Shared autonomy has played a major role in assistive mobile robotics as it has the potential to effectively balance user satisfaction and smooth functioning of systems by adapting itself to each user's needs and preferences. Many shared control paradigms have been developed over the years. However, despite these advancements, shared control paradigms have not been widely adopted as there are several integral aspects that have not fully matured. The purpose of this paper is to discuss and review various aspects of shared control and the technologies leading up to the current advancements in shared control for assistive mobile robots. METHODS A comprehensive review of the literature was conducted following a dichotomy of studies from the pre-2000 and the post-2000 periods to focus on both the early developments and the current state of the art in this domain. RESULTS A systematic review of 135 research papers and 7 review papers selected from the literature was conducted. To facilitate the organization of the reviewed work, a 6-level ladder categorization was developed based on the extent of autonomy shared between the human and the robot in the use of assistive mobile robots. This taxonomy highlights the chronological improvements in this domain. CONCLUSION It was found that most prior studies have focussed on basic functionalities, thus paving the way for research to now focus on the higher levels of the ladder taxonomy. It was concluded that further research in the domain must focus on ensuring safety in mobility and adaptability to varying environments.Implications for rehabilitationShared autonomy in assistive mobile robots plays a vital role in effectively adapting to ensure safety while also considering the user comfort.User's immediate desires should be considered in decision making to ensure that the users are in control of the assistive robots.The current focus of research should be towards successful adaptation of the assistive mobile robots to varying environments to assure safety of the user.
Collapse
Affiliation(s)
- Sumukha Udupa
- Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI, USA.,Robotics Institute, University of Michigan, Ann Arbor, MI, USA
| | - Vineet R Kamat
- Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI, USA.,Robotics Institute, University of Michigan, Ann Arbor, MI, USA
| | - Carol C Menassa
- Department of Civil and Environmental Engineering, University of Michigan, Ann Arbor, MI, USA.,Robotics Institute, University of Michigan, Ann Arbor, MI, USA
| |
Collapse
|
19
|
Trambaiolli LR, Tiwari A, Falk TH. Affective Neurofeedback Under Naturalistic Conditions: A Mini-Review of Current Achievements and Open Challenges. FRONTIERS IN NEUROERGONOMICS 2021; 2:678981. [PMID: 38235228 PMCID: PMC10790905 DOI: 10.3389/fnrgo.2021.678981] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 03/10/2021] [Accepted: 04/28/2021] [Indexed: 01/19/2024]
Abstract
Affective neurofeedback training allows for the self-regulation of the putative circuits of emotion regulation. This approach has recently been studied as a possible additional treatment for psychiatric disorders, presenting positive effects in symptoms and behaviors. After neurofeedback training, a critical aspect is the transference of the learned self-regulation strategies to outside the laboratory and how to continue reinforcing these strategies in non-controlled environments. In this mini-review, we discuss the current achievements of affective neurofeedback under naturalistic setups. For this, we first provide a brief overview of the state-of-the-art for affective neurofeedback protocols. We then discuss virtual reality as a transitional step toward the final goal of "in-the-wild" protocols and current advances using mobile neurotechnology. Finally, we provide a discussion of open challenges for affective neurofeedback protocols in-the-wild, including topics such as convenience and reliability, environmental effects in attention and workload, among others.
Collapse
Affiliation(s)
- Lucas R. Trambaiolli
- Basic Neuroscience Division, McLean Hospital–Harvard Medical School, Belmont, MA, United States
| | - Abhishek Tiwari
- Institut National de la Recherche Scientifique, University of Quebec, Montreal, QC, Canada
| | - Tiago H. Falk
- Institut National de la Recherche Scientifique, University of Quebec, Montreal, QC, Canada
| |
Collapse
|
20
|
Cao L, Li G, Xu Y, Zhang H, Shu X, Zhang D. A brain-actuated robotic arm system using non-invasive hybrid brain-computer interface and shared control strategy. J Neural Eng 2021; 18. [PMID: 33862607 DOI: 10.1088/1741-2552/abf8cb] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2020] [Accepted: 04/16/2021] [Indexed: 01/20/2023]
Abstract
Objective.The electroencephalography (EEG)-based brain-computer interfaces (BCIs) have been used in the control of robotic arms. The performance of non-invasive BCIs may not be satisfactory due to the poor quality of EEG signals, so the shared control strategies were tried as an alternative solution. However, most of the existing shared control methods set the arbitration rules manually, which highly depended on the specific tasks and developer's experience. In this study, we proposed a novel shared control model that automatically optimized the control commands in a dynamical way based on the context in real-time control. Besides, we employed the hybrid BCI to better allocate commands with multiple functions. The system allowed non-invasive BCI users to manipulate a robotic arm moving in a three-dimensional (3D) space and complete a pick-place task of multiple objects.Approach.Taking the scene information obtained by computer vision as a knowledge base, a machine agent was designed to infer the user's intention and generate automatic commands. Based on the inference confidence and user's characteristic, the proposed shared control model fused the machine autonomy and human intention dynamically for robotic arm motion optimization during the online control. In addition, we introduced a hybrid BCI scheme that applied steady-state visual evoked potentials and motor imagery to the divided primary and secondary BCI interfaces to better allocate the BCI resources (e.g. decoding computing power, screen occupation) and realize the multi-dimensional control of the robotic arm.Main results.Eleven subjects participated in the online experiments of picking and placing five objects that scattered at different positions in a 3D workspace. The results showed that most of the subjects could control the robotic arm to complete accurate and robust picking task with an average success rate of approximately 85% under the shared control strategy, while the average success rate of placing task controlled by pure BCI was 50% approximately.Significance.In this paper, we proposed a novel shared controller for motion automatic optimization, together with a hybrid BCI control scheme that allocated paradigms according to the importance of commands to realize multi-dimensional and effective control of a robotic arm. Our study indicated that the shared control strategy with hybrid BCI could greatly improve the performance of the brain-actuated robotic arm system.
Collapse
Affiliation(s)
- Linfeng Cao
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Guangye Li
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Yang Xu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Heng Zhang
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Xiaokang Shu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Dingguo Zhang
- Department of Electronic and Electrical Engineering, University of Bath, Bath, United Kingdom
| |
Collapse
|
21
|
Iwane F, Iturrate I, Chavarriaga R, Millán JDR. Invariability of EEG error-related potentials during continuous feedback protocols elicited by erroneous actions at predicted or unpredicted states. J Neural Eng 2021; 18. [PMID: 33882461 DOI: 10.1088/1741-2552/abfa70] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2020] [Accepted: 04/21/2021] [Indexed: 11/11/2022]
Abstract
Objective.When humans perceive an erroneous action, an EEG error-related potential (ErrP) is elicited as a neural response. ErrPs have been largely investigated in discrete feedback protocols, where actions are executed at discrete steps, to enable seamless brain-computer interaction. However, there are only a few studies that investigate ErrPs in continuous feedback protocols. The objective of the present study is to better understand the differences between two types of ErrPs elicited during continuous feedback protocols, where errors may occur either at predicted or unpredicted states. We hypothesize that ErrPs of the unpredicted state is associated with longer latency as it requires higher cognitive workload to evaluate actions compared to the predicted states.Approach.Participants monitored the trajectory of an autonomous cursor that occasionally made erroneous actions on its way to the target in two conditions, namely, predicted or unpredicted states. After characterizing the ErrP waveform elicited by erroneous actions in the two conditions, we performed single-trial decoding of ErrPs in both synchronous (i.e. time-locked to the onset of the erroneous action) and asynchronous manner. Furthermore, we explored the possibility to transfer decoders built with data of one of the conditions to the other condition.Main results.As hypothesized, erroneous actions at unpredicted states gave rise to ErrPs with higher latency than erroneous actions at predicted states, a correlate of higher cognitive effort in the former condition. Moreover, ErrP decoders trained in a given condition successfully transferred to the other condition with a slight loss of classification performance. This was the case for synchronous as well as asynchronous ErrP decoding, showing the invariability of ErrPs across conditions.Significance.These results advance the characterization of ErrPs during continuous feedback protocols, enlarging the potential use of ErrPs during natural operation of brain-controlled devices as it is not necessary to have different decoders for each kind of erroneous conditions.
Collapse
Affiliation(s)
- Fumiaki Iwane
- Learning Algorithms and Systems Laboratory (LASA) , École Polytechnique Féderale de Lausanne (EPFL), 1015 Lausanne, Switzerland.,Department of Electrical and Computer Engineering , The University of Texas at Austin, Austin, TX 78712, United States of America
| | - Iñaki Iturrate
- École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,Amazon , Barcelona, Spain
| | - Ricardo Chavarriaga
- École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,ZHAW Datalab , Zurich University of Applied Sciences, Winterthur, Switzerland
| | - José Del R Millán
- Department of Electrical and Computer Engineering , The University of Texas at Austin, Austin, TX 78712, United States of America.,École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,Department of Neurology , The University of Texas at Austin, Austin, TX 78712, United States of America
| |
Collapse
|
22
|
Abstract
The prospect and potentiality of interfacing minds with machines has long captured human imagination. Recent advances in biomedical engineering, computer science, and neuroscience are making brain–computer interfaces a reality, paving the way to restoring and potentially augmenting human physical and mental capabilities. Applications of brain–computer interfaces are being explored in applications as diverse as security, lie detection, alertness monitoring, gaming, education, art, and human cognition augmentation. The present tutorial aims to survey the principal features and challenges of brain–computer interfaces (such as reliable acquisition of brain signals, filtering and processing of the acquired brainwaves, ethical and legal issues related to brain–computer interface (BCI), data privacy, and performance assessment) with special emphasis to biomedical engineering and automation engineering applications. The content of this paper is aimed at students, researchers, and practitioners to glimpse the multifaceted world of brain–computer interfacing.
Collapse
|
23
|
Saha S, Mamun KA, Ahmed K, Mostafa R, Naik GR, Darvishi S, Khandoker AH, Baumert M. Progress in Brain Computer Interface: Challenges and Opportunities. Front Syst Neurosci 2021; 15:578875. [PMID: 33716680 PMCID: PMC7947348 DOI: 10.3389/fnsys.2021.578875] [Citation(s) in RCA: 74] [Impact Index Per Article: 24.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/01/2020] [Accepted: 01/06/2021] [Indexed: 12/13/2022] Open
Abstract
Brain computer interfaces (BCI) provide a direct communication link between the brain and a computer or other external devices. They offer an extended degree of freedom either by strengthening or by substituting human peripheral working capacity and have potential applications in various fields such as rehabilitation, affective computing, robotics, gaming, and neuroscience. Significant research efforts on a global scale have delivered common platforms for technology standardization and help tackle highly complex and non-linear brain dynamics and related feature extraction and classification challenges. Time-variant psycho-neurophysiological fluctuations and their impact on brain signals impose another challenge for BCI researchers to transform the technology from laboratory experiments to plug-and-play daily life. This review summarizes state-of-the-art progress in the BCI field over the last decades and highlights critical challenges.
Collapse
Affiliation(s)
- Simanto Saha
- School of Electrical and Electronic Engineering, The University of Adelaide, Adelaide, SA, Australia
- Department of Electrical and Electronic Engineering, United International University, Dhaka, Bangladesh
| | - Khondaker A. Mamun
- Advanced Intelligent Multidisciplinary Systems (AIMS) Lab, Department of Computer Science and Engineering, United International University, Dhaka, Bangladesh
| | - Khawza Ahmed
- Department of Electrical and Electronic Engineering, United International University, Dhaka, Bangladesh
| | - Raqibul Mostafa
- Department of Electrical and Electronic Engineering, United International University, Dhaka, Bangladesh
| | - Ganesh R. Naik
- Adelaide Institute for Sleep Health, College of Medicine and Public Health, Flinders University, Adelaide, SA, Australia
| | - Sam Darvishi
- School of Electrical and Electronic Engineering, The University of Adelaide, Adelaide, SA, Australia
| | - Ahsan H. Khandoker
- Healthcare Engineering Innovation Center, Department of Biomedical Engineering, Khalifa University of Science and Technology, Abu Dhabi, United Arab Emirates
| | - Mathias Baumert
- School of Electrical and Electronic Engineering, The University of Adelaide, Adelaide, SA, Australia
| |
Collapse
|
24
|
Ben Khelifa MM, Lamti HA, Hugel V. A Muscular and Cerebral Physiological Indices Assessment for Stress Measuring during Virtual Wheelchair Guidance. Brain Sci 2021; 11:274. [PMID: 33671722 PMCID: PMC7926415 DOI: 10.3390/brainsci11020274] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/02/2021] [Revised: 02/10/2021] [Accepted: 02/14/2021] [Indexed: 11/24/2022] Open
Abstract
The work presented in this manuscript has the purpose to assess the relationship between human factors and physiological indices. We discuss the relationship between stress as human factor and cerebral and muscular signals as features. Ten male paraplegic, right-handed subjects were volunteers for the experiment (mean age 34 ±6). They drove a virtual wheelchair in an indoor environment. They filled five missions where, in each one, an environmental parameter was changed. Meanwhile, they were equipped with Electromyography (EMG) sensors and Electroencephalography (EEG). Frequency and temporal features were filtered and extracted. Principal component analysis (PCA), Fisher's tests, repeated measure Anova and post hoc Tukey test (α = 0.05) were implemented for statistics. Environmental modifications are subject to induce stress, which impacts muscular and cerebral activities. While the time pressure parameter was the most influent, the transition from static to moving obstacles (avatars), tends to have a significant impact on stress levels. However, adding more moving obstacles did not show any impact. A synchronization factor was noticed between cerebral and muscular features in higher stress levels. Further examination is needed to assess EEG reliability in these situations.
Collapse
Affiliation(s)
- Mohamed Moncef Ben Khelifa
- Impact de l’Activite Physique sur la Sante (IAPS) Laboratory, South University, 83130 Toulon-Var, France
| | - Hachem A. Lamti
- Conception de Systemes Mecaniques et Robotiques (COSMER) Laboratory, South University, 83130 Toulon-Var, France; (H.A.L.); (V.H.)
| | - Vincent Hugel
- Conception de Systemes Mecaniques et Robotiques (COSMER) Laboratory, South University, 83130 Toulon-Var, France; (H.A.L.); (V.H.)
| |
Collapse
|
25
|
Ko LW, D SVS, Huang Y, Lu YC, Shaw S, Jung TP. SSVEP-assisted RSVP Brain-Computer Interface paradigm for multi-target classification. J Neural Eng 2020; 18. [PMID: 33291083 DOI: 10.1088/1741-2552/abd1c0] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2020] [Accepted: 12/08/2020] [Indexed: 11/12/2022]
Abstract
Brain-Computer Interface (BCI) is actively involved in optimizing the communication medium between the human brain and external devices. OBJECTIVE Rapid serial visual presentation (RSVP) is a robust and highly efficient BCI technique in recognizing target objects but suffers from limited target selections. The BCI systems that combine steady-state visual evoked potential (SSVEP) and RSVP can mitigate this limitation and allow users to operate on multiple targets. APPROACH This study proposes a novel SSVEP-assisted RSVP BCI model to improve the performance of classifying the target/non-target objects in a multi-target scenario. In this paradigm, SSVEP stimuli helps in identifying the user's focus location and RSVP stimuli that elicits event-related potentials (ERPs) differentiate target and non-target objects. MAIN RESULTS The proposed model achieved an offline accuracy of 81.59% by using 12 electroencephalogram (EEG) channels and an online (real-time) accuracy of 78.10% when only 4 EEG channels are considered. Further, the biomarkers of physiological states are analyzed to assess the cognitive states (mental fatigue and user attention) of the participants based on resting theta and alpha band powers. The results indicate an inverse relationship between the BCI performance and the resting EEG power, validating that the subjects' performance is affected by physiological states for prolonged BCI tasks. SIGNIFICANCE Our findings demonstrate that the combination of SSVEP and RSVP stimuli improves the BCI performance and further enhances the possibility of performing multiple user command tasks, which are inevitable in real-world applications. Additionally, the cognitive state biomarkers discussed imply the need for an efficient and attractive experimental paradigm that reduces the physiological state disparities and provide enhanced BCI performance.
Collapse
Affiliation(s)
- Li-Wei Ko
- Institute of Bioinformatics and Systems Biology, National Chiao Tung University, BA308, No. 75, Boai St., Hsinchu City, Hsinchu, 300, TAIWAN
| | - Sandeep Vara Sankar D
- International PhD Program in Interdisciplinary Neuroscience, Department of Biological Science and Technology, National Chiao Tung University, BA306, No. 75, Boai Street Hsinchu City, Hsinchu, 300, TAIWAN
| | - Yufei Huang
- University of Texas at San Antonio Department of Electrical and Computer Engineering, One UTSA Circle San Antonio, TX, San Antonio, Texas, 78249-0669, UNITED STATES
| | - Yun-Chen Lu
- Department of Biological Science and Technology, National Chiao Tung University, BA306, No. 75, Boai St., Hsinchu City, Hsinchu, 300, TAIWAN
| | - Siddharth Shaw
- Department of Biological Science and Technology, National Chiao Tung University, BA306, No. 75, Boai St., Hsinchu City, Hsinchu, 300, TAIWAN
| | - Tzyy-Ping Jung
- Institute for Neural Computation and Institute of Engineering in Medicine, University of California - San Diego, 9500 Gilman Drive, La Jolla, CA, CA, 92093-0559, UNITED STATES
| |
Collapse
|
26
|
Plug-and-play supervisory control using muscle and brain signals for real-time gesture and error detection. Auton Robots 2020. [DOI: 10.1007/s10514-020-09916-x] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
AbstractEffective human supervision of robots can be key for ensuring correct robot operation in a variety of potentially safety-critical scenarios. This paper takes a step towards fast and reliable human intervention in supervisory control tasks by combining two streams of human biosignals: muscle and brain activity acquired via EMG and EEG, respectively. It presents continuous classification of left and right hand-gestures using muscle signals, time-locked classification of error-related potentials using brain signals (unconsciously produced when observing an error), and a framework that combines these pipelines to detect and correct robot mistakes during multiple-choice tasks. The resulting hybrid system is evaluated in a “plug-and-play” fashion with 7 untrained subjects supervising an autonomous robot performing a target selection task. Offline analysis further explores the EMG classification performance, and investigates methods to select subsets of training data that may facilitate generalizable plug-and-play classifiers.
Collapse
|
27
|
Li H, Bi L, Shi H. Modeling of Human Operator Behavior for Brain-Actuated Mobile Robots Steering. IEEE Trans Neural Syst Rehabil Eng 2020; 28:2063-2072. [PMID: 32746321 DOI: 10.1109/tnsre.2020.3009376] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Human operator control of brain-actuated robot steering based on electroencephalograph (EEG)-signals is a complex behavior consisting of surroundings perceiving, decision making, and commands issuing and differs among individual operators. However, no existing models allow decoupling the user from the loop to improve the system design and testing process, which can capture such behavior of a brain-actuated robot. To address this problem, in this paper, we propose an operator brain-controlled steering model consisting of an operator decision model based on the queuing network (QN) cognitive architecture and a brain-machine interface (BMI) performance model. The QN-based operator decision model can mimic the human decision process with the individual operator differences considered. The new BMI performance model is built to represent the varied accuracy of BMI during brain-controlled direction operations. Furthermore, the model is simulated and validated against the results of human operator-in-the-loop experiments. The results show that the proposed model can reproduce the behavior of human operators thanks to its similar direction control performance.
Collapse
|
28
|
Pan H, Mi W, Lei X, Zhong W. A closed-loop BMI system design based on the improved SJIT model and the network of Izhikevich neurons. Neurocomputing 2020. [DOI: 10.1016/j.neucom.2020.03.047] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/01/2023]
|
29
|
Deng X, Liang Yu Z, Lin C, Gu Z, Li Y. Self-adaptive shared control with brain state evaluation network for human-wheelchair cooperation. J Neural Eng 2020; 17:045005. [PMID: 32413885 DOI: 10.1088/1741-2552/ab937e] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
OBJECTIVE For the shared control systems, how to trade off the control weight between robot autonomy and human operator is an important issue, especially for BCI-based systems. However, most of existing shared controllers have paid less attention to the effects caused by subjects with different levels of brain control ability. APPROACH In this paper, a brain state evaluation network, termed BSE-NET, is proposed to evaluate subjects' brain control ability online based on quantized attention-gated kernel reinforcement learning. With the output of BSE-NET (confidence score), a shared controller is designed to dynamically adjust the control weight between robot autonomy and human operator. MAIN RESULTS The experimental results show that most of subjects achieved high and stable experimental success rate of approximately 90%. Furthermore, for subjects with different accuracy on EEG decoding, a proper confidence score can be dynamically generated to reflect their levels of brain control ability, and the proposed system can effectively adjust the control weight in all-time shared control. SIGNIFICANCE We discuss how our proposed method shows promise for BCI applications that can evaluate subjects' brain control ability online as well as provide a method for the research on self-adaptive shared control to adaptively balance control weight between subject's instruction and robot autonomy.
Collapse
Affiliation(s)
- Xiaoyan Deng
- School of Automation Science and Engineering, South China University of Technology, Guangzhou 510641, People's Republic of China. Pazhou Lab, Guangzhou 510335, People's Republic of China
| | | | | | | | | |
Collapse
|
30
|
Ortiz M, Iáñez E, Gaxiola-Tirado JA, Gutiérrez D, Azorín JM. Study of the Functional Brain Connectivity and Lower-Limb Motor Imagery Performance After Transcranial Direct Current Stimulation. Int J Neural Syst 2020; 30:2050038. [DOI: 10.1142/s0129065720500380] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
Abstract
The use of transcranial direct current stimulation (tDCS) has been related to the improvement of motor and learning tasks. The current research studies the effects of an asymmetric tDCS setup over brain connectivity, when the subject is performing a motor imagery (MI) task during five consecutive days. A brain–computer interface (BCI) based on electroencephalography is simulated in offline analysis to study the effect that tDCS has over different electrode configurations for the BCI. This way, the BCI performance is used as a validation index of the effect of the tDCS setup by the analysis of the classifier accuracy of the experimental sessions. In addition, the relationship between the brain connectivity and the BCI accuracy performance is analyzed. Results indicate that tDCS group, in comparison to the placebo sham group, shows a higher significant number of connectivity interactions in the motor electrodes during MI tasks and an increasing BCI accuracy over the days. However, the asymmetric tDCS setup does not improve the BCI performance of the electrodes in the intended hemisphere.
Collapse
Affiliation(s)
- Mario Ortiz
- Brain-Machine Interface Systems Lab, Miguel Hernández University of Elche, Avenida Universidad sn. Ed. Innova, Elche, Alicante 03202, Spain
| | - Eduardo Iáñez
- Brain-Machine Interface Systems Lab, Miguel Hernández University of Elche, Avenida Universidad sn. Ed. Innova, Elche, Alicante 03202, Spain
| | - Jorge A. Gaxiola-Tirado
- Center for Research and Advanced Studies (Cinvestav), Monterrey’s Unit, Vía del Conocimiento 201 PIIT, 66600, Apodaca NL 66600, Mexico
| | - David Gutiérrez
- Center for Research and Advanced Studies (Cinvestav), Monterrey’s Unit, Vía del Conocimiento 201 PIIT, 66600, Apodaca NL 66600, Mexico
| | - José M. Azorín
- Systems Engineering and Automation Department, Miguel Hernández University of Elche, Avenida Universidad sn. Ed. Innova, Elche, Alicante 03202, Spain
| |
Collapse
|
31
|
Comprehensive review on brain-controlled mobile robots and robotic arms based on electroencephalography signals. INTEL SERV ROBOT 2020. [DOI: 10.1007/s11370-020-00328-5] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/25/2022]
|
32
|
Rashid M, Sulaiman N, P P Abdul Majeed A, Musa RM, Ab Nasir AF, Bari BS, Khatun S. Current Status, Challenges, and Possible Solutions of EEG-Based Brain-Computer Interface: A Comprehensive Review. Front Neurorobot 2020; 14:25. [PMID: 32581758 PMCID: PMC7283463 DOI: 10.3389/fnbot.2020.00025] [Citation(s) in RCA: 102] [Impact Index Per Article: 25.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2019] [Accepted: 04/08/2020] [Indexed: 12/12/2022] Open
Abstract
Brain-Computer Interface (BCI), in essence, aims at controlling different assistive devices through the utilization of brain waves. It is worth noting that the application of BCI is not limited to medical applications, and hence, the research in this field has gained due attention. Moreover, the significant number of related publications over the past two decades further indicates the consistent improvements and breakthroughs that have been made in this particular field. Nonetheless, it is also worth mentioning that with these improvements, new challenges are constantly discovered. This article provides a comprehensive review of the state-of-the-art of a complete BCI system. First, a brief overview of electroencephalogram (EEG)-based BCI systems is given. Secondly, a considerable number of popular BCI applications are reviewed in terms of electrophysiological control signals, feature extraction, classification algorithms, and performance evaluation metrics. Finally, the challenges to the recent BCI systems are discussed, and possible solutions to mitigate the issues are recommended.
Collapse
Affiliation(s)
- Mamunur Rashid
- Faculty of Electrical & Electronics Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Norizam Sulaiman
- Faculty of Electrical & Electronics Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Anwar P P Abdul Majeed
- Innovative Manufacturing, Mechatronics and Sports Laboratory, Faculty of Manufacturing and Mechatronic Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Rabiu Muazu Musa
- Centre for Fundamental and Continuing Education, Universiti Malaysia Terengganu, Kuala Nerus, Malaysia
| | - Ahmad Fakhri Ab Nasir
- Innovative Manufacturing, Mechatronics and Sports Laboratory, Faculty of Manufacturing and Mechatronic Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Bifta Sama Bari
- Faculty of Electrical & Electronics Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| | - Sabira Khatun
- Faculty of Electrical & Electronics Engineering Technology, Universiti Malaysia Pahang, Pekan, Malaysia
| |
Collapse
|
33
|
Benitez-Andonegui A, Burden R, Benning R, Möckel R, Lührs M, Sorger B. An Augmented-Reality fNIRS-Based Brain-Computer Interface: A Proof-of-Concept Study. Front Neurosci 2020; 14:346. [PMID: 32410938 PMCID: PMC7199634 DOI: 10.3389/fnins.2020.00346] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/03/2019] [Accepted: 03/23/2020] [Indexed: 02/04/2023] Open
Abstract
Augmented reality (AR) enhances the user's environment by projecting virtual objects into the real world in real-time. Brain-computer interfaces (BCIs) are systems that enable users to control external devices with their brain signals. BCIs can exploit AR technology to interact with the physical and virtual world and to explore new ways of displaying feedback. This is important for users to perceive and regulate their brain activity or shape their communication intentions while operating in the physical world. In this study, twelve healthy participants were introduced to and asked to choose between two motor-imagery tasks: mental drawing and interacting with a virtual cube. Participants first performed a functional localizer run, which was used to select a single fNIRS channel for decoding their intentions in eight subsequent choice-encoding runs. In each run participants were asked to select one choice of a six-item list. A rotating AR cube was displayed on a computer screen as the main stimulus, where each face of the cube was presented for 6 s and represented one choice of the six-item list. For five consecutive trials, participants were instructed to perform the motor-imagery task when the face of the cube that represented their choice was facing them (therewith temporally encoding the selected choice). In the end of each run, participants were provided with the decoded choice based on a joint analysis of all five trials. If the decoded choice was incorrect, an active error-correction procedure was applied by the participant. The choice list provided in each run was based on the decoded choice of the previous run. The experimental design allowed participants to navigate twice through a virtual menu that consisted of four levels if all choices were correctly decoded. Here we demonstrate for the first time that by using AR feedback and flexible choice encoding in form of search trees, we can increase the degrees of freedom of a BCI system. We also show that participants can successfully navigate through a nested menu and achieve a mean accuracy of 74% using a single motor-imagery task and a single fNIRS channel.
Collapse
Affiliation(s)
- Amaia Benitez-Andonegui
- Department Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht Brain Imaging Center, Maastricht University, Maastricht, Netherlands
- Laboratory for Cognitive Robotics and Complex Self-Organizing Systems, Department of Data Science and Knowledge Engineering, Faculty of Science and Engineering, Maastricht University, Maastricht, Netherlands
| | - Rodion Burden
- Department Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht Brain Imaging Center, Maastricht University, Maastricht, Netherlands
| | - Richard Benning
- Instrumentation Engineering, Dean and Directors Office, Faculty of Psychology and Neuroscience, Maastricht University, Maastricht, Netherlands
| | - Rico Möckel
- Laboratory for Cognitive Robotics and Complex Self-Organizing Systems, Department of Data Science and Knowledge Engineering, Faculty of Science and Engineering, Maastricht University, Maastricht, Netherlands
| | - Michael Lührs
- Department Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht Brain Imaging Center, Maastricht University, Maastricht, Netherlands
- Research Department, Brain Innovation B.V., Maastricht, Netherlands
| | - Bettina Sorger
- Department Cognitive Neuroscience, Faculty of Psychology and Neuroscience, Maastricht Brain Imaging Center, Maastricht University, Maastricht, Netherlands
| |
Collapse
|
34
|
Allison BZ, Kübler A, Jin J. 30+ years of P300 brain-computer interfaces. Psychophysiology 2020; 57:e13569. [PMID: 32301143 DOI: 10.1111/psyp.13569] [Citation(s) in RCA: 30] [Impact Index Per Article: 7.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/07/2019] [Revised: 01/07/2020] [Accepted: 01/20/2020] [Indexed: 11/28/2022]
Abstract
Brain-computer interfaces (BCIs) directly measure brain activity with no physical movement and translate the neural signals into messages. BCIs that employ the P300 event-related brain potential often have used the visual modality. The end user is presented with flashing stimuli that indicate selections for communication, control, or both. Counting each flash that corresponds to a specific target selection while ignoring other flashes will elicit P300s to only the target selection. P300 BCIs also have been implemented using auditory or tactile stimuli. P300 BCIs have been used with a variety of applications for severely disabled end users in their homes without frequent expert support. P300 BCI research and development has made substantial progress, but challenges remain before these tools can become practical devices for impaired patients and perhaps healthy people.
Collapse
Affiliation(s)
- Brendan Z Allison
- Cognitive Science Department, University of California at San Diego, La Jolla, CA, USA
| | - Andrea Kübler
- Psychology Department, University of Würzburg, Würzburg, Germany
| | - Jing Jin
- Key Laboratory of Advanced Control and Optimization for Chemical Processes, Ministry of Education, East China University of Science and Technology, Shanghai, P.R. China
| |
Collapse
|
35
|
Kolkhorst H, Veit J, Burgard W, Tangermann M. A Robust Screen-Free Brain-Computer Interface for Robotic Object Selection. Front Robot AI 2020; 7:38. [PMID: 33501206 PMCID: PMC7806045 DOI: 10.3389/frobt.2020.00038] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/02/2019] [Accepted: 03/05/2020] [Indexed: 11/13/2022] Open
Abstract
Brain signals represent a communication modality that can allow users of assistive robots to specify high-level goals, such as the object to fetch and deliver. In this paper, we consider a screen-free Brain-Computer Interface (BCI), where the robot highlights candidate objects in the environment using a laser pointer, and the user goal is decoded from the evoked responses in the electroencephalogram (EEG). Having the robot present stimuli in the environment allows for more direct commands than traditional BCIs that require the use of graphical user interfaces. Yet bypassing a screen entails less control over stimulus appearances. In realistic environments, this leads to heterogeneous brain responses for dissimilar objects-posing a challenge for reliable EEG classification. We model object instances as subclasses to train specialized classifiers in the Riemannian tangent space, each of which is regularized by incorporating data from other objects. In multiple experiments with a total of 19 healthy participants, we show that our approach not only increases classification performance but is also robust to both heterogeneous and homogeneous objects. While especially useful in the case of a screen-free BCI, our approach can naturally be applied to other experimental paradigms with potential subclass structure.
Collapse
Affiliation(s)
- Henrich Kolkhorst
- Brain State Decoding Lab, Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
- Autonomous Intelligent Systems, Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
- Cluster of Excellence BrainLinks-BrainTools, University of Freiburg, Freiburg im Breisgau, Germany
| | - Joseline Veit
- Brain State Decoding Lab, Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
| | - Wolfram Burgard
- Autonomous Intelligent Systems, Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
- Cluster of Excellence BrainLinks-BrainTools, University of Freiburg, Freiburg im Breisgau, Germany
- Toyota Research Institute, Los Altos, CA, United States
| | - Michael Tangermann
- Brain State Decoding Lab, Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
- Autonomous Intelligent Systems, Department of Computer Science, University of Freiburg, Freiburg im Breisgau, Germany
- Cluster of Excellence BrainLinks-BrainTools, University of Freiburg, Freiburg im Breisgau, Germany
| |
Collapse
|
36
|
Iturrate I, Chavarriaga R, Millán JDR. General principles of machine learning for brain-computer interfacing. HANDBOOK OF CLINICAL NEUROLOGY 2020; 168:311-328. [PMID: 32164862 DOI: 10.1016/b978-0-444-63934-9.00023-8] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Abstract
Brain-computer interfaces (BCIs) are systems that translate brain activity patterns into commands that can be executed by an artificial device. This enables the possibility of controlling devices such as a prosthetic arm or exoskeleton, a wheelchair, typewriting applications, or games directly by modulating our brain activity. For this purpose, BCI systems rely on signal processing and machine learning algorithms to decode the brain activity. This chapter provides an overview of the main steps required to do such a process, including signal preprocessing, feature extraction and selection, and decoding. Given the large amount of possible methods that can be used for these processes, a comprehensive review of them is beyond the scope of this chapter, and it is focused instead on the general principles that should be taken into account, as well as discussing good practices on how these methods should be applied and evaluated for proper design of reliable BCI systems.
Collapse
Affiliation(s)
- Iñaki Iturrate
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Ricardo Chavarriaga
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland; Institute of Applied Information Technology (InIT), Zurich University of Applied Sciences ZHAW, Winterthur, Switzerland.
| | - José Del R Millán
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, United States; Department of Neurology, The University of Texas at Austin, Austin, TX, United States
| |
Collapse
|
37
|
A Novel P300 Classification Algorithm Based on a Principal Component Analysis-Convolutional Neural Network. APPLIED SCIENCES-BASEL 2020. [DOI: 10.3390/app10041546] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Aiming at enhancing the classification accuracy of P300 Electroencephalogram signals in a non-invasive brain–computer interface system, a novel P300 electroencephalogram signals classification algorithm is proposed which is based on improved convolutional neural network. In the data preprocessing part, the proposed P300 classification algorithm used the Principal Component Analysis algorithm to not only remove the noise and artifacts in the data, but also increase the data processing speed. Furthermore, the proposed P300 classification algorithm employed the parallel convolution method to improve the traditional convolutional neural network framework, which can increase the network depth and improve the network’s ability to classify P300 electroencephalogram signals. The proposed algorithm was evaluated by two datasets (the dataset from the competition and the dataset from the laboratory). The results show that, in the dataset I, the proposed P300 classification algorithm could obtain accuracy rates higher than 95%, and achieve one of the best performances in four classification algorithms, while, in the dataset II, the proposed P300 classification algorithm can get accuracy rates higher than 90%, and is superior to the other three algorithms in all ten subjects. These demonstrated the effectiveness of the proposed algorithm. The proposed classification algorithm can be applied in the actual brain–computer interface systems to help people with disability in the daily lives.
Collapse
|
38
|
Wirth C, Toth J, Arvaneh M. "You Have Reached Your Destination": A Single Trial EEG Classification Study. Front Neurosci 2020; 14:66. [PMID: 32116513 PMCID: PMC7027274 DOI: 10.3389/fnins.2020.00066] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2019] [Accepted: 01/16/2020] [Indexed: 01/24/2023] Open
Abstract
Studies have established that it is possible to differentiate between the brain's responses to observing correct and incorrect movements in navigation tasks. Furthermore, these classifications can be used as feedback for a learning-based BCI, to allow real or virtual robots to find quasi-optimal routes to a target. However, when navigating it is important not only to know we are moving in the right direction toward a target, but also to know when we have reached it. We asked participants to observe a virtual robot performing a 1-dimensional navigation task. We recorded EEG and then performed neurophysiological analysis on the responses to two classes of correct movements: those that moved closer to the target but did not reach it, and those that did reach the target. Further, we used a stepwise linear classifier on time-domain features to differentiate the classes on a single-trial basis. A second data set was also used to further test this single-trial classification. We found that the amplitude of the P300 was significantly greater in cases where the movement reached the target. Interestingly, we were able to classify the EEG signals evoked when observing the two classes of correct movements against each other with mean overall accuracy of 66.5 and 68.0% for the two data sets, with greater than chance levels of accuracy achieved for all participants. As a proof of concept, we have shown that it is possible to classify the EEG responses in observing these different correct movements against each other using single-trial EEG. This could be used as part of a learning-based BCI and opens a new door toward a more autonomous BCI navigation system.
Collapse
Affiliation(s)
- Christopher Wirth
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, United Kingdom
| | - Jake Toth
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, United Kingdom
| | - Mahnaz Arvaneh
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, United Kingdom
| |
Collapse
|
39
|
Tonin L, Bauer FC, del R. Millan J. The Role of the Control Framework for Continuous Teleoperation of a Brain–Machine Interface-Driven Mobile Robot. IEEE T ROBOT 2020. [DOI: 10.1109/tro.2019.2943072] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
40
|
Virgilio G. CD, Sossa A. JH, Antelis JM, Falcón LE. Spiking Neural Networks applied to the classification of motor tasks in EEG signals. Neural Netw 2020; 122:130-143. [DOI: 10.1016/j.neunet.2019.09.037] [Citation(s) in RCA: 44] [Impact Index Per Article: 11.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/28/2019] [Revised: 09/17/2019] [Accepted: 09/23/2019] [Indexed: 11/29/2022]
|
41
|
Ratcliffe L, Puthusserypady S. Importance of Graphical User Interface in the design of P300 based Brain–Computer Interface systems. Comput Biol Med 2020; 117:103599. [DOI: 10.1016/j.compbiomed.2019.103599] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/18/2019] [Revised: 12/12/2019] [Accepted: 12/29/2019] [Indexed: 12/01/2022]
|
42
|
Quiles E, Suay F, Candela G, Chio N, Jiménez M, Álvarez-Kurogi L. Low-Cost Robotic Guide Based on a Motor Imagery Brain-Computer Interface for Arm Assisted Rehabilitation. INTERNATIONAL JOURNAL OF ENVIRONMENTAL RESEARCH AND PUBLIC HEALTH 2020; 17:ijerph17030699. [PMID: 31973155 PMCID: PMC7036782 DOI: 10.3390/ijerph17030699] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/23/2019] [Revised: 01/10/2020] [Accepted: 01/17/2020] [Indexed: 12/26/2022]
Abstract
Motor imagery has been suggested as an efficient alternative to improve the rehabilitation process of affected limbs. In this study, a low-cost robotic guide is implemented so that linear position can be controlled via the user’s motor imagination of movement intention. The patient can use this device to move the arm attached to the guide according to their own intentions. The first objective of this study was to check the feasibility and safety of the designed robotic guide controlled via a motor imagery (MI)-based brain–computer interface (MI-BCI) in healthy individuals, with the ultimate aim to apply it to rehabilitation patients. The second objective was to determine which are the most convenient MI strategies to control the different assisted rehabilitation arm movements. The results of this study show a better performance when the BCI task is controlled with an action–action MI strategy versus an action–relaxation one. No statistically significant difference was found between the two action–action MI strategies.
Collapse
Affiliation(s)
- Eduardo Quiles
- Instituto de Automática e Informática Industrial, Universitat Politècnica de València, 46022 València, Spain;
- Correspondence: ; Tel.: +34-96-387-7007 (ext. 75793)
| | - Ferran Suay
- Departament de Psicobiologia, Facultat de Psicologia, Universitat de València, 46010 València, Spain; (F.S.); (G.C.)
| | - Gemma Candela
- Departament de Psicobiologia, Facultat de Psicologia, Universitat de València, 46010 València, Spain; (F.S.); (G.C.)
| | - Nayibe Chio
- Instituto de Automática e Informática Industrial, Universitat Politècnica de València, 46022 València, Spain;
- Facultad de Ingeniería, Ingeniería Mecatrónica, Universidad Autónoma de Bucaramanga, Bucaramanga 680003, Colombia
| | - Manuel Jiménez
- Facultad de Educación, Universidad Internacional de la Rioja, 26006 Logroño, Spain; (M.J.); (L.Á.-K.)
| | - Leandro Álvarez-Kurogi
- Facultad de Educación, Universidad Internacional de la Rioja, 26006 Logroño, Spain; (M.J.); (L.Á.-K.)
| |
Collapse
|
43
|
Liu M, Wang K, Chen X, Zhao J, Chen Y, Wang H, Wang J, Xu S. Indoor Simulated Training Environment for Brain-Controlled Wheelchair Based on Steady-State Visual Evoked Potentials. Front Neurorobot 2020; 13:101. [PMID: 31998108 PMCID: PMC6961652 DOI: 10.3389/fnbot.2019.00101] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/19/2019] [Accepted: 11/19/2019] [Indexed: 12/29/2022] Open
Abstract
Brain-controlled wheelchair (BCW) has the potential to improve the quality of life for people with motor disabilities. A lot of training is necessary for users to learn and improve BCW control ability and the performances of BCW control are crucial for patients in daily use. In consideration of safety and efficiency, an indoor simulated training environment is built up in this paper to improve the performance of BCW control. The indoor simulated environment mainly realizes BCW implementation, simulated training scenario setup, path planning and recommendation, simulated operation, and scoring. And the BCW is based on steady-state visual evoked potentials (SSVEP) and the filter bank canonical correlation analysis (FBCCA) is used to analyze the electroencephalography (EEG). Five tasks include individual accuracy, simple linear path, obstacles avoidance, comprehensive steering scenarios, and evaluation task are designed, 10 healthy subjects were recruited and carried out the 7-days training experiment to assess the performance of the training environment. Scoring and command-consuming are conducted to evaluate the improvement before and after training. The results indicate that the average accuracy is 93.55% and improves from 91.05% in the first stage to 96.05% in the second stage (p = 0.001). Meanwhile, the average score increases from 79.88 in the first session to 96.66 in the last session and tend to be stable (p < 0.001). The average number of commands and collisions to complete the tasks decreases significantly with or without the approximate shortest path (p < 0.001). These results show that the performance of subjects in BCW control achieves improvement and verify the feasibility and effectiveness of the proposed simulated training environment.
Collapse
Affiliation(s)
- Ming Liu
- Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin, China
| | - Kangning Wang
- Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin, China.,School of Electronics and Information Engineering, Tianjin Polytechnic University, Tianjin, China
| | - Xiaogang Chen
- Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin, China
| | - Jing Zhao
- Institute of Traditional Chinese Medicine, Tianjin University of Traditional Chinese Medicine, Tianjin, China
| | - Yuanyuan Chen
- School of Microelectronics, Tianjin University, Tianjin, China
| | - Huiquan Wang
- School of Electronics and Information Engineering, Tianjin Polytechnic University, Tianjin, China
| | - Jinhai Wang
- School of Electronics and Information Engineering, Tianjin Polytechnic University, Tianjin, China
| | - Shengpu Xu
- Institute of Biomedical Engineering, Chinese Academy of Medical Sciences and Peking Union Medical College, Tianjin, China
| |
Collapse
|
44
|
Deng X, Yu ZL, Lin C, Gu Z, Li Y. A Bayesian Shared Control Approach for Wheelchair Robot With Brain Machine Interface. IEEE Trans Neural Syst Rehabil Eng 2019; 28:328-338. [PMID: 31825869 DOI: 10.1109/tnsre.2019.2958076] [Citation(s) in RCA: 23] [Impact Index Per Article: 4.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
To enhance the performance of the brain-actuated robot system, a novel shared controller based on Bayesian approach is proposed for intelligently combining robot automatic control and brain-actuated control, which takes into account the uncertainty of robot perception, action and human control. Based on maximum a posteriori probability (MAP), this method establishes the probabilistic models of human and robot control commands to realize the optimal control of a brain-actuated shared control system. Application on an intelligent Bayesian shared control system based on steady-state visual evoked potential (SSVEP)-based brain machine interface (BMI) is presented for all-time continuous wheelchair navigation task. Moreover, to obtain more accurate brain control commands for shared controller and adapt the proposed system to the uncertainty of electroencephalogram (EEG), a hierarchical brain control mechanism with feedback rule is designed. Experiments have been conducted to verify the proposed system in several scenarios. Eleven subjects participated in our experiments and the results illustrate the effectiveness of the proposed method.
Collapse
|
45
|
Nagarajan G, Minu R, Jayanthiladevi A. Brain computer interface for smart hardware device. INTERNATIONAL JOURNAL OF RF TECHNOLOGIES 2019. [DOI: 10.3233/rft-180167] [Citation(s) in RCA: 26] [Impact Index Per Article: 5.2] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Affiliation(s)
- G. Nagarajan
- Department of CSE, Sathyabama Institute of Science and Technology, Chennai, Tamil Nadu, India
| | - R.I. Minu
- Department of CSE, SRM Institute of Science and Technology, Kattankulathur, Tamil Nadu, India
| | | |
Collapse
|
46
|
Li Z, Li J, Zhao S, Yuan Y, Kang Y, Chen CLP. Adaptive Neural Control of a Kinematically Redundant Exoskeleton Robot Using Brain-Machine Interfaces. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2019; 30:3558-3571. [PMID: 30346293 DOI: 10.1109/tnnls.2018.2872595] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/08/2023]
Abstract
In this paper, a closed-loop control has been developed for the exoskeleton robot system based on brain-machine interface (BMI). Adaptive controllers in joint space, a redundancy resolution method at the velocity level, and commands that generated from BMI in task space have been integrated effectively to make the robot perform manipulation tasks controlled by human operator's electroencephalogram. By extracting the features from neural activity, the proposed intention decoding algorithm can generate the commands to control the exoskeleton robot. To achieve optimal motion, a redundancy resolution at the velocity level has been implemented through neural dynamics optimization. Considering human-robot interaction force as well as coupled dynamics during the exoskeleton operation, an adaptive controller with redundancy resolution has been designed to drive the exoskeleton tracking the planned trajectory in human brain and to offer a convenient method of dynamics compensation with minimal knowledge of the dynamics parameters of the exoskeleton robot. Extensive experiments which employed a few subjects have been carried out. In the experiments, subjects successfully fulfilled the given manipulation tasks with convergence of tracking errors, which verified that the proposed brain-controlled exoskeleton robot system is effective.
Collapse
|
47
|
Robot Motion Control via an EEG-Based Brain–Computer Interface by Using Neural Networks and Alpha Brainwaves. ELECTRONICS 2019. [DOI: 10.3390/electronics8121387] [Citation(s) in RCA: 21] [Impact Index Per Article: 4.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 01/05/2023]
Abstract
Modern achievements accomplished in both cognitive neuroscience and human–machine interaction technologies have enhanced the ability to control devices with the human brain by using Brain–Computer Interface systems. Particularly, the development of brain-controlled mobile robots is very important because systems of this kind can assist people, suffering from devastating neuromuscular disorders, move and thus improve their quality of life. The research work presented in this paper, concerns the development of a system which performs motion control in a mobile robot in accordance to the eyes’ blinking of a human operator via a synchronous and endogenous Electroencephalography-based Brain–Computer Interface, which uses alpha brain waveforms. The received signals are filtered in order to extract suitable features. These features are fed as inputs to a neural network, which is properly trained in order to properly guide the robotic vehicle. Experimental tests executed on 12 healthy subjects of various gender and age, proved that the system developed is able to perform movements of the robotic vehicle, under control, in forward, left, backward, and right direction according to the alpha brainwaves of its operator, with an overall accuracy equal to 92.1%.
Collapse
|
48
|
Li Z, Yuan Y, Luo L, Su W, Zhao K, Xu C, Huang J, Pi M. Hybrid Brain/Muscle Signals Powered Wearable Walking Exoskeleton Enhancing Motor Ability in Climbing Stairs Activity. ACTA ACUST UNITED AC 2019. [DOI: 10.1109/tmrb.2019.2949865] [Citation(s) in RCA: 79] [Impact Index Per Article: 15.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
49
|
Combined Sensing, Cognition, Learning, and Control for Developing Future Neuro-Robotics Systems: A Survey. IEEE Trans Cogn Dev Syst 2019. [DOI: 10.1109/tcds.2019.2897618] [Citation(s) in RCA: 13] [Impact Index Per Article: 2.6] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
50
|
Velasco-Álvarez F, Sancha-Ros S, García-Garaluz E, Fernández-Rodríguez Á, Medina-Juliá MT, Ron-Angevin R. UMA-BCI Speller: An easily configurable P300 speller tool for end users. COMPUTER METHODS AND PROGRAMS IN BIOMEDICINE 2019; 172:127-138. [PMID: 30902124 DOI: 10.1016/j.cmpb.2019.02.015] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/19/2018] [Revised: 02/08/2019] [Accepted: 02/27/2019] [Indexed: 06/09/2023]
Abstract
BACKGROUND AND OBJECTIVE Some neurodegenerative conditions can severely limit patients' capability to communicate because of the loss of muscular control. Brain-computer interfaces may help in the restoration of communication with these patients, bypassing the muscular activity, so that brain signals can be directly interpreted by a computer. There are many studies regarding brain-controlled spellers; however, these systems do not usually leap out of the lab because of technical and economic requirements. As a consequence, the potential end users do not benefit from these scientific advances in their daily life. The objective of this paper is to present a novel brain-controlled speller designed to be used by patients due to its versatility and ease of use. METHODS The brain-computer interface research group of the University of Málaga (UMA-BCI) has developed a speller application based on the well-known P300 potential which can be easily installed, configured and used. The application supports the common P300 paradigms: the Row-Column Paradigm and the Rapid Serial Visual Presentation Paradigm. The inner core of the application is implemented with a widely used and studied platform, BCI2000, which ensures its reliability and allows other researchers to apply modifications at will in order to test new features. Ten naïve volunteers carried out exercises using the application and completed usability tests for evaluation purposes. RESULTS New subjects using the application managed to set up and use the proposed speller in less than an hour. The positive results of the evaluation through the usability tests support this application's ease of use. CONCLUSIONS A new brain-controlled spelling tool has been presented whose aim is to be used by severely paralyzed patients in their daily lives, as well as by researchers to test new spelling features.
Collapse
Affiliation(s)
| | - Salvador Sancha-Ros
- ENESO Tecnología de Adaptación S.L., Parque Tecnológico de Andalucía, Málaga, Spain
| | | | | | | | - Ricardo Ron-Angevin
- Departamento de Tecnología Electrónica, Universidad de Málaga, 29071 Málaga, Spain.
| |
Collapse
|