1
|
Ren G, Kumar A, Mahmoud SS, Fang Q. A deep neural network and transfer learning combined method for cross-task classification of error-related potentials. Front Hum Neurosci 2024; 18:1394107. [PMID: 38933146 PMCID: PMC11199896 DOI: 10.3389/fnhum.2024.1394107] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/01/2024] [Accepted: 05/22/2024] [Indexed: 06/28/2024] Open
Abstract
Background Error-related potentials (ErrPs) are electrophysiological responses that naturally occur when humans perceive wrongdoing or encounter unexpected events. It offers a distinctive means of comprehending the error-processing mechanisms within the brain. A method for detecting ErrPs with high accuracy holds significant importance for various ErrPs-based applications, such as human-in-the-loop Brain-Computer Interface (BCI) systems. Nevertheless, current methods fail to fulfill the generalization requirements for detecting such ErrPs due to the high non-stationarity of EEG signals across different tasks and the limited availability of ErrPs datasets. Methods This study introduces a deep learning-based model that integrates convolutional layers and transformer encoders for the classification of ErrPs. Subsequently, a model training strategy, grounded in transfer learning, is proposed for the effective training of the model. The datasets utilized in this study are available for download from the publicly accessible databases. Results In cross-task classification, an average accuracy of about 78% was achieved, exceeding the baseline. Furthermore, in the leave-one-subject-out, within-session, and cross-session classification scenarios, the proposed model outperformed the existing techniques with an average accuracy of 71.81, 78.74, and 77.01%, respectively. Conclusions Our approach contributes to mitigating the challenge posed by limited datasets in the ErrPs field, achieving this by reducing the requirement for extensive training data for specific target tasks. This may serve as inspiration for future studies that concentrate on ErrPs and their applications.
Collapse
Affiliation(s)
| | | | | | - Qiang Fang
- Department of Biomedical Engineering, Shantou University, Shantou, China
| |
Collapse
|
2
|
Iwane F, Porssut T, Blanke O, Chavarriaga R, Del R Millán J, Herbelin B, Boulic R. Customizing the human-avatar mapping based on EEG error related potentials. J Neural Eng 2024; 21:026016. [PMID: 38386506 DOI: 10.1088/1741-2552/ad2c02] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/03/2023] [Accepted: 01/26/2024] [Indexed: 02/24/2024]
Abstract
Objective.A key challenge of virtual reality (VR) applications is to maintain a reliable human-avatar mapping. Users may lose the sense of controlling (sense of agency), owning (sense of body ownership), or being located (sense of self-location) inside the virtual body when they perceive erroneous interaction, i.e. a break-in-embodiment (BiE). However, the way to detect such an inadequate event is currently limited to questionnaires or spontaneous reports from users. The ability to implicitly detect BiE in real-time enables us to adjust human-avatar mapping without interruption.Approach.We propose and empirically demonstrate a novel brain computer interface (BCI) approach that monitors the occurrence of BiE based on the users' brain oscillatory activity in real-time to adjust the human-avatar mapping in VR. We collected EEG activity of 37 participants while they performed reaching movements with their avatar with different magnitude of distortion.Main results.Our BCI approach seamlessly predicts occurrence of BiE in varying magnitude of erroneous interaction. The mapping has been customized by BCI-reinforcement learning (RL) closed-loop system to prevent BiE from occurring. Furthermore, a non-personalized BCI decoder generalizes to new users, enabling 'Plug-and-Play' ErrP-based non-invasive BCI. The proposed VR system allows customization of human-avatar mapping without personalized BCI decoders or spontaneous reports.Significance.We anticipate that our newly developed VR-BCI can be useful to maintain an engaging avatar-based interaction and a compelling immersive experience while detecting when users notice a problem and seamlessly correcting it.
Collapse
Affiliation(s)
- Fumiaki Iwane
- Learning Algorithms and Systems Laboratory (LASA), École Polytechnique Féderale de Lausanne (EPFL), 1015 Lausanne, Switzerland
- Dept. of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, United States of America
- Dept. of Neurology, The University of Texas at Austin, Austin, TX 78712, United States of America
| | - Thibault Porssut
- Immersive Interaction Research Group (IIG), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- Laboratory of Cognitive Neuroscience (LNCO), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- Capgemini Engineering, Paris, France
| | - Olaf Blanke
- Laboratory of Cognitive Neuroscience (LNCO), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
- Dept. of Neurology, Geneva University Hospitals, Geneva, Switzerland
| | - Ricardo Chavarriaga
- Centre for Artificial Intelligence, Zurich University of Applied Sciences (ZHAW), Winterthur, Switzerland
| | - José Del R Millán
- Dept. of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, United States of America
- Dept. of Neurology, The University of Texas at Austin, Austin, TX 78712, United States of America
- Dept. of Biomedical Engineering, The University of Texas at Austin, Austin, TX 78712, United States of America
- Mulva Clinic for the Neurosciences, The University of Texas at Austin, Austin, TX 78712, United States of America
| | - Bruno Herbelin
- Laboratory of Cognitive Neuroscience (LNCO), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Ronan Boulic
- Immersive Interaction Research Group (IIG), Ecole Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| |
Collapse
|
3
|
John AR, Singh AK, Gramann K, Liu D, Lin CT. Prediction of cognitive conflict during unexpected robot behavior under different mental workload conditions in a physical human-robot collaboration. J Neural Eng 2024; 21:026010. [PMID: 38295415 DOI: 10.1088/1741-2552/ad2494] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2023] [Accepted: 01/31/2024] [Indexed: 02/02/2024]
Abstract
Objective. Brain-computer interface (BCI) technology is poised to play a prominent role in modern work environments, especially a collaborative environment where humans and machines work in close proximity, often with physical contact. In a physical human robot collaboration (pHRC), the robot performs complex motion sequences. Any unexpected robot behavior or faulty interaction might raise safety concerns. Error-related potentials, naturally generated by the brain when a human partner perceives an error, have been extensively employed in BCI as implicit human feedback to adapt robot behavior to facilitate a safe and intuitive interaction. However, the integration of BCI technology with error-related potential for robot control demands failure-free integration of highly uncertain electroencephalography (EEG) signals, particularly influenced by the physical and cognitive state of the user. As a higher workload on the user compromises their access to cognitive resources needed for error awareness, it is crucial to study how mental workload variations impact the error awareness as it might raise safety concerns in pHRC. In this study, we aim to study how cognitive workload affects the error awareness of a human user engaged in a pHRC.Approach. We designed a blasting task with an abrasive industrial robot and manipulated the mental workload with a secondary arithmetic task of varying difficulty. EEG data, perceived workload, task and physical performance were recorded from 24 participants moving the robot arm. The error condition was achieved by the unexpected stopping of the robot in 33% of trials.Main results. We observed a diminished amplitude for the prediction error negativity (PEN) and error positivity (Pe), indicating reduced error awareness with increasing mental workload. We further observed an increased frontal theta power and increasing trend in the central alpha and central beta power after the unexpected robot stopping compared to when the robot stopped correctly at the target. We also demonstrate that a popular convolution neural network model, EEGNet, could predict the amplitudes of PEN and Pe from the EEG data prior to the error.Significance. This prediction model could be instrumental in developing an online prediction model that could forewarn the system and operators of the diminished error awareness of the user, alluding to a potential safety breach in error-related potential-based BCI system for pHRC. Therefore, our work paves the way for embracing BCI technology in pHRC to optimally adapt the robot behavior for personalized user experience using real-time brain activity, enriching the quality of the interaction.
Collapse
Affiliation(s)
- Alka Rachel John
- Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia
| | - Avinash K Singh
- Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia
| | - Klaus Gramann
- Department of Biological Psychology and Neuroergonomics, TU Berlin, Berlin, Germany
| | - Dikai Liu
- Robotics Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia
| | - Chin-Teng Lin
- Australian Artificial Intelligence Institute, Faculty of Engineering and Information Technology, University of Technology Sydney, Sydney, Australia
| |
Collapse
|
4
|
Kueper N, Chari K, Bütefür J, Habenicht J, Rossol T, Kim SK, Tabie M, Kirchner F, Kirchner EA. EEG and EMG dataset for the detection of errors introduced by an active orthosis device. Front Hum Neurosci 2024; 18:1304311. [PMID: 38317650 PMCID: PMC10839100 DOI: 10.3389/fnhum.2024.1304311] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/29/2023] [Accepted: 01/08/2024] [Indexed: 02/07/2024] Open
Affiliation(s)
- Niklas Kueper
- Robotics Innovation Centre, German Research Centre for Artificial Intelligence (DFKI), Bremen, Germany
| | - Kartik Chari
- Robotics Innovation Centre, German Research Centre for Artificial Intelligence (DFKI), Bremen, Germany
| | - Judith Bütefür
- Institute of Medical Technology Systems, University of Duisburg-Essen, Duisburg, Germany
| | - Julia Habenicht
- Institute of Medical Technology Systems, University of Duisburg-Essen, Duisburg, Germany
| | - Tobias Rossol
- Robotics Innovation Centre, German Research Centre for Artificial Intelligence (DFKI), Bremen, Germany
| | - Su Kyoung Kim
- Robotics Innovation Centre, German Research Centre for Artificial Intelligence (DFKI), Bremen, Germany
| | - Marc Tabie
- Robotics Innovation Centre, German Research Centre for Artificial Intelligence (DFKI), Bremen, Germany
| | - Frank Kirchner
- Robotics Innovation Centre, German Research Centre for Artificial Intelligence (DFKI), Bremen, Germany
- Robotics Lab, University of Bremen, Bremen, Germany
| | - Elsa Andrea Kirchner
- Robotics Innovation Centre, German Research Centre for Artificial Intelligence (DFKI), Bremen, Germany
- Institute of Medical Technology Systems, University of Duisburg-Essen, Duisburg, Germany
| |
Collapse
|
5
|
Luo R, Mai X, Meng J. Effect of motion state variability on error-related potentials during continuous feedback paradigms and their consequences for classification. J Neurosci Methods 2024; 401:109982. [PMID: 37839711 DOI: 10.1016/j.jneumeth.2023.109982] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/05/2023] [Revised: 09/11/2023] [Accepted: 10/11/2023] [Indexed: 10/17/2023]
Abstract
BACKGROUND An erroneous motion would elicit the error-related potential (ErrP) when humans monitor the behavior of the external devices. This EEG modality has been largely applied to brain-computer interface in an active or passive manner with discrete visual feedback. However, the effect of variable motion state on ErrP morphology and classification performance raises concerns when the interaction is conducted with continuous visual feedback. NEW METHOD In the present study, we designed a cursor control experiment. Participants monitored a continuously moving cursor to reach the target on one side of the screen. Motion state varied multiple times with two factors: (1) motion direction and (2) motion speed. The effects of these two factors on the morphological characteristics and classification performance of ErrP were analyzed. Furthermore, an offline simulation was performed to evaluate the effectiveness of the proposed extended ErrP-decoder in resolving the interference by motion direction changes. RESULTS The statistical analyses revealed that motion direction and motion speed significantly influenced the amplitude of feedback-ERN and frontal-Pe components, while only motion direction significantly affected the classification performance. COMPARISON WITH EXISTING METHODS Significant deviation was found in ErrP detection utilizing classical correct-versus-erroneous event training. However, this bias can be alleviated by 16% by the extended ErrP-decoder. CONCLUSION The morphology and classification performance of ErrP signal can be affected by motion state variability during continuous feedback paradigms. The results enhance the comprehension of ErrP morphological components and shed light on the detection of BCI's error behavior in practical continuous control.
Collapse
Affiliation(s)
- Ruijie Luo
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Ximing Mai
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China
| | - Jianjun Meng
- Department of Mechanical Engineering, Shanghai Jiao Tong University, Shanghai, China; State Key Laboratory of Mechanical System and Vibration, Shanghai Jiao Tong University, Shanghai, China.
| |
Collapse
|
6
|
Gomez-Andres A, Cerda-Company X, Cucurell D, Cunillera T, Rodríguez-Fornells A. Decoding agency attribution using single trial error-related brain potentials. Psychophysiology 2024; 61:e14434. [PMID: 37668293 DOI: 10.1111/psyp.14434] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2023] [Revised: 07/25/2023] [Accepted: 08/08/2023] [Indexed: 09/06/2023]
Abstract
Being able to distinguish between self and externally generated actions is a key factor influencing learning and adaptive behavior. Previous literature has highlighted that whenever a person makes or perceives an error, a series of error-related potentials (ErrPs) can be detected in the electroencephalographic (EEG) signal, such as the error-related negativity (ERN) component. Recently, ErrPs have gained a lot of interest for the use in brain-computer interface (BCI) applications, which give the user the ability to communicate by means of decoding his/her brain activity. Here, we explored the feasibility of employing a support vector machine classifier to accurately disentangle self-agency errors from other-agency errors from the EEG signal at a single-trial level in a sample of 23 participants. Our results confirmed the viability of correctly disentangling self/internal versus other/external agency-error attributions at different stages of brain processing based on the latency and the spatial topographical distribution of key ErrP features, namely, the ERN and P600 components, respectively. These results offer a new perspective on how to distinguish self versus externally generated errors providing new potential implementations on BCI systems.
Collapse
Affiliation(s)
- Alba Gomez-Andres
- Cognition and Brain Plasticity Group [Bellvitge Biomedical Research Institute-IDIBELL], Barcelona, Spain
- Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain
| | - Xim Cerda-Company
- Cognition and Brain Plasticity Group [Bellvitge Biomedical Research Institute-IDIBELL], Barcelona, Spain
- Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain
- Universitat Autonoma de Barcelona, Barcelona, Spain
| | - David Cucurell
- Cognition and Brain Plasticity Group [Bellvitge Biomedical Research Institute-IDIBELL], Barcelona, Spain
- Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain
| | - Toni Cunillera
- Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain
| | - Antoni Rodríguez-Fornells
- Cognition and Brain Plasticity Group [Bellvitge Biomedical Research Institute-IDIBELL], Barcelona, Spain
- Department of Cognition, Development and Educational Psychology, University of Barcelona, Barcelona, Spain
- Institute of Neurosciences (UBNeuro), University of Barcelona, Barcelona, Spain
- Catalan Institution for Research and Advanced Studies, ICREA, Barcelona, Spain
| |
Collapse
|
7
|
Kim SK, Kirchner EA. Detection of tactile-based error-related potentials (ErrPs) in human-robot interaction. Front Neurorobot 2023; 17:1297990. [PMID: 38162893 PMCID: PMC10756909 DOI: 10.3389/fnbot.2023.1297990] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/20/2023] [Accepted: 11/22/2023] [Indexed: 01/03/2024] Open
Abstract
Robot learning based on implicitly extracted error detections (e.g., EEG-based error detections) has been well-investigated in human-robot interaction (HRI). In particular, the use of error-related potential (ErrP) evoked when recognizing errors is advantageous for robot learning when evaluation criteria cannot be explicitly defined, e.g., due to the complex behavior of robots. In most studies, erroneous behavior of robots were recognized visually. In some studies, visuo-tactile stimuli were used to evoke ErrPs or a tactile cue was used to indicate upcoming errors. To our knowledge, there are no studies in which ErrPs are evoked when recognizing errors only via the tactile channel. Hence, we investigated ErrPs evoked by tactile recognition of errors during HRI. In our scenario, subjects recognized errors caused by incorrect behavior of an orthosis during the execution of arm movements tactilely. EEG data from eight subjects was recorded. Subjects were asked to give a motor response to ensure error detection. Latency between the occurrence of errors and the response to errors was expected to be short. We assumed that the motor related brain activity is timely correlated with the ErrP and might be used from the classifier. To better interpret and test our results, we therefore tested ErrP detections in two additional scenarios, i.e., without motor response and with delayed motor response. In addition, we transferred three scenarios (motor response, no motor response, delayed motor response). Response times to error was short. However, high ErrP-classification performance was found for all subjects in case of motor response and no motor response condition. Further, ErrP classification performance was reduced for the transfer between motor response and delayed motor response, but not for the transfer between motor response and no motor response. We have shown that tactilely induced errors can be detected with high accuracy from brain activity. Our preliminary results suggest that also in tactile ErrPs the brain response is clear enough such that motor response is not relevant for classification. However, in future work, we will more systematically investigate tactile-based ErrP classification.
Collapse
Affiliation(s)
- Su Kyoung Kim
- Robotics Innovation Center, German Research Center for Artificial Intelligence GmbH, Bremen, Germany
| | - Elsa Andrea Kirchner
- Robotics Innovation Center, German Research Center for Artificial Intelligence GmbH, Bremen, Germany
- Institute of Medical Technology Systems, University of Duisburg-Essen, Duisburg, Germany
| |
Collapse
|
8
|
Vukelić M, Bui M, Vorreuther A, Lingelbach K. Combining brain-computer interfaces with deep reinforcement learning for robot training: a feasibility study in a simulation environment. FRONTIERS IN NEUROERGONOMICS 2023; 4:1274730. [PMID: 38234482 PMCID: PMC10790930 DOI: 10.3389/fnrgo.2023.1274730] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Accepted: 10/31/2023] [Indexed: 01/19/2024]
Abstract
Deep reinforcement learning (RL) is used as a strategy to teach robot agents how to autonomously learn complex tasks. While sparsity is a natural way to define a reward in realistic robot scenarios, it provides poor learning signals for the agent, thus making the design of good reward functions challenging. To overcome this challenge learning from human feedback through an implicit brain-computer interface (BCI) is used. We combined a BCI with deep RL for robot training in a 3-D physical realistic simulation environment. In a first study, we compared the feasibility of different electroencephalography (EEG) systems (wet- vs. dry-based electrodes) and its application for automatic classification of perceived errors during a robot task with different machine learning models. In a second study, we compared the performance of the BCI-based deep RL training to feedback explicitly given by participants. Our findings from the first study indicate the use of a high-quality dry-based EEG-system can provide a robust and fast method for automatically assessing robot behavior using a sophisticated convolutional neural network machine learning model. The results of our second study prove that the implicit BCI-based deep RL version in combination with the dry EEG-system can significantly accelerate the learning process in a realistic 3-D robot simulation environment. Performance of the BCI-based trained deep RL model was even comparable to that achieved by the approach with explicit human feedback. Our findings emphasize the usage of BCI-based deep RL methods as a valid alternative in those human-robot applications where no access to cognitive demanding explicit human feedback is available.
Collapse
Affiliation(s)
- Mathias Vukelić
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| | - Michael Bui
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| | - Anna Vorreuther
- Applied Neurocognitive Systems, Institute of Human Factors and Technology Management (IAT), University of Stuttgart, Stuttgart, Germany
| | - Katharina Lingelbach
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| |
Collapse
|
9
|
Iwane F, Billard A, Millán JDR. Inferring individual evaluation criteria for reaching trajectories with obstacle avoidance from EEG signals. Sci Rep 2023; 13:20163. [PMID: 37978205 PMCID: PMC10656489 DOI: 10.1038/s41598-023-47136-2] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2023] [Accepted: 11/09/2023] [Indexed: 11/19/2023] Open
Abstract
During reaching actions, the human central nerve system (CNS) generates the trajectories that optimize effort and time. When there is an obstacle in the path, we make sure that our arm passes the obstacle with a sufficient margin. This comfort margin varies between individuals. When passing a fragile object, risk-averse individuals may adopt a larger margin by following the longer path than risk-prone people do. However, it is not known whether this variation is associated with a personalized cost function used for the individual optimal control policies and how it is represented in our brain activity. This study investigates whether such individual variations in evaluation criteria during reaching results from differentiated weighting given to energy minimization versus comfort, and monitors brain error-related potentials (ErrPs) evoked when subjects observe a robot moving dangerously close to a fragile object. Seventeen healthy participants monitored a robot performing safe, daring and unsafe trajectories around a wine glass. Each participant displayed distinct evaluation criteria on the energy efficiency and comfort of robot trajectories. The ErrP-BCI outputs successfully inferred such individual variation. This study suggests that ErrPs could be used in conjunction with an optimal control approach to identify the personalized cost used by CNS. It further opens new avenues for the use of brain-evoked potential to train assistive robotic devices through the use of neuroprosthetic interfaces.
Collapse
Affiliation(s)
- Fumiaki Iwane
- Learning Algorithms and Systems Laboratory (LASA), École Polytechnique Fédérale de Lausanne (EPFL), 1015, Lausanne, Switzerland.
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, 78712, USA.
- Department of Neurology, The University of Texas at Austin, Austin, TX, 78712, USA.
| | - Aude Billard
- Learning Algorithms and Systems Laboratory (LASA), École Polytechnique Fédérale de Lausanne (EPFL), 1015, Lausanne, Switzerland
| | - José Del R Millán
- Chandra Family Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, 78712, USA
- Department of Neurology, The University of Texas at Austin, Austin, TX, 78712, USA
- Department of Biomedical Engineering, The University of Texas at Austin, Austin, TX, 78712, USA
- Mulva Clinic for the Neurosciences, The University of Texas at Austin, Austin, TX, 78712, USA
| |
Collapse
|
10
|
Wirth C, Toth J, Arvaneh M. Bayesian learning from multi-way EEG feedback for robot navigation and target identification. Sci Rep 2023; 13:16925. [PMID: 37805540 PMCID: PMC10560278 DOI: 10.1038/s41598-023-44077-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/25/2023] [Accepted: 10/03/2023] [Indexed: 10/09/2023] Open
Abstract
Many brain-computer interfaces require a high mental workload. Recent research has shown that this could be greatly alleviated through machine learning, inferring user intentions via reactive brain responses. These signals are generated spontaneously while users merely observe assistive robots performing tasks. Using reactive brain signals, existing studies have addressed robot navigation tasks with a very limited number of potential target locations. Moreover, they use only binary, error-vs-correct classification of robot actions, leaving more detailed information unutilised. In this study a virtual robot had to navigate towards, and identify, target locations in both small and large grids, wherein any location could be the target. For the first time, we apply a system utilising detailed EEG information: 4-way classification of movements is performed, including specific information regarding when the target is reached. Additionally, we classify whether targets are correctly identified. Our proposed Bayesian strategy infers the most likely target location from the brain's responses. The experimental results show that our novel use of detailed information facilitates a more efficient and robust system than the state-of-the-art. Furthermore, unlike state-of-the-art approaches, we show scalability of our proposed approach: By tuning parameters appropriately, our strategy correctly identifies 98% of targets, even in large search spaces.
Collapse
Affiliation(s)
- Christopher Wirth
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, S1 4DT, UK.
- School of Medical Sciences, University of Manchester, Manchester, M13 9NT, UK.
| | - Jake Toth
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, S1 4DT, UK
| | - Mahnaz Arvaneh
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, S1 4DT, UK
| |
Collapse
|
11
|
Ye J, Collinger JL, Wehbe L, Gaunt R. Neural Data Transformer 2: Multi-context Pretraining for Neural Spiking Activity. BIORXIV : THE PREPRINT SERVER FOR BIOLOGY 2023:2023.09.18.558113. [PMID: 37781630 PMCID: PMC10541112 DOI: 10.1101/2023.09.18.558113] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/03/2023]
Abstract
The neural population spiking activity recorded by intracortical brain-computer interfaces (iBCIs) contain rich structure. Current models of such spiking activity are largely prepared for individual experimental contexts, restricting data volume to that collectable within a single session and limiting the effectiveness of deep neural networks (DNNs). The purported challenge in aggregating neural spiking data is the pervasiveness of context-dependent shifts in the neural data distributions. However, large scale unsupervised pretraining by nature spans heterogeneous data, and has proven to be a fundamental recipe for successful representation learning across deep learning. We thus develop Neural Data Transformer 2 (NDT2), a spatiotemporal Transformer for neural spiking activity, and demonstrate that pretraining can leverage motor BCI datasets that span sessions, subjects, and experimental tasks. NDT2 enables rapid adaptation to novel contexts in downstream decoding tasks and opens the path to deployment of pretrained DNNs for iBCI control. Code: https://github.com/joel99/context_general_bci.
Collapse
Affiliation(s)
- Joel Ye
- Rehab Neural Engineering Labs, University of Pittsburgh
- Neuroscience Institute, Carnegie Mellon University
- Center for the Neural Basis of Cognition, Pittsburgh
| | - Jennifer L. Collinger
- Rehab Neural Engineering Labs, University of Pittsburgh
- Center for the Neural Basis of Cognition, Pittsburgh
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh
- Department of Bioengineering, University of Pittsburgh
- Department of Biomedical Engineering, Carnegie Mellon University
| | - Leila Wehbe
- Neuroscience Institute, Carnegie Mellon University
- Center for the Neural Basis of Cognition, Pittsburgh
- Machine Learning Department, Carnegie Mellon University
| | - Robert Gaunt
- Rehab Neural Engineering Labs, University of Pittsburgh
- Center for the Neural Basis of Cognition, Pittsburgh
- Department of Physical Medicine and Rehabilitation, University of Pittsburgh
- Department of Bioengineering, University of Pittsburgh
- Department of Biomedical Engineering, Carnegie Mellon University
| |
Collapse
|
12
|
Iwane F, Sobolewski A, Chavarriaga R, Millán JDR. EEG error-related potentials encode magnitude of errors and individual perceptual thresholds. iScience 2023; 26:107524. [PMID: 37636067 PMCID: PMC10448161 DOI: 10.1016/j.isci.2023.107524] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/14/2023] [Revised: 06/15/2023] [Accepted: 07/28/2023] [Indexed: 08/29/2023] Open
Abstract
Error-related potentials (ErrPs) are a prominent electroencephalogram (EEG) correlate of performance monitoring, and so crucial for learning and adapting our behavior. It is poorly understood whether ErrPs encode further information beyond error awareness. We report an experiment with sixteen participants over three sessions in which occasional visual rotations of varying magnitude occurred during a cursor reaching task. We designed a brain-computer interface (BCI) to detect ErrPs that provided real-time feedback. The individual ErrP-BCI decoders exhibited good transfer across sessions and scalability over the magnitude of errors. A non-linear relationship between the ErrP-BCI output and the magnitude of errors predicts individual perceptual thresholds to detect errors. We also reveal theta-gamma oscillatory coupling that co-varied with the magnitude of the required adjustment. Our findings open new avenues to probe and extend current theories of performance monitoring by incorporating continuous human interaction tasks and analysis of the ErrP complex rather than individual peaks.
Collapse
Affiliation(s)
- Fumiaki Iwane
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
- Department of Neurology, The University of Texas at Austin, Austin, TX 78712, USA
- Learning Algorithms and Systems Laboratory (LASA), École Polytechnique Fédérale de Lausanne (EPFL), 1015 Lausanne, Switzerland
| | - Aleksander Sobolewski
- Wyss Center for Bio and Neuroengineering, Campus Biotech, 1202 Genève, Switzerland
- École Polytechnique Fédérale de Lausanne (EPFL), Campus Biotech, 1202 Genève, Switzerland
| | - Ricardo Chavarriaga
- École Polytechnique Fédérale de Lausanne (EPFL), Campus Biotech, 1202 Genève, Switzerland
- Centre for Artificial Intelligence, Zurich University of Applied Sciences (ZHAW), 8401 Winterthur, Switzerland
| | - José del R. Millán
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712, USA
- Department of Neurology, The University of Texas at Austin, Austin, TX 78712, USA
- École Polytechnique Fédérale de Lausanne (EPFL), Campus Biotech, 1202 Genève, Switzerland
- Mulva Clinic for the Neurosciences, The University of Texas at Austin, Austin, TX 78712, USA
| |
Collapse
|
13
|
Iwane F, Dash D, Salamanca-Giron RF, Hayward W, Bönstrup M, Buch ER, Cohen LG. Combined low-frequency brain oscillatory activity and behavior predict future errors in human motor skill. Curr Biol 2023; 33:3145-3154.e5. [PMID: 37442139 DOI: 10.1016/j.cub.2023.06.040] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/10/2023] [Revised: 03/24/2023] [Accepted: 06/13/2023] [Indexed: 07/15/2023]
Abstract
Human skills are composed of sequences of individual actions performed with utmost precision. When occasional errors occur, they may have serious consequences, for example, when pilots are manually landing a plane. In such cases, the ability to predict an error before it occurs would clearly be advantageous. Here, we asked whether it is possible to predict future errors in a keyboard procedural human motor skill. We report that prolonged keypress transition times (KTTs), reflecting slower speed, and anomalous delta-band oscillatory activity in cingulate-entorhinal-precuneus brain regions precede upcoming errors in skill. Combined anomalous low-frequency activity and prolonged KTTs predicted up to 70% of future errors. Decoding strength (posterior probability of error) increased progressively approaching the errors. We conclude that it is possible to predict future individual errors in skill sequential performance.
Collapse
Affiliation(s)
- Fumiaki Iwane
- Human Cortical Physiology and Neurorehabilitation Section, NINDS, NIH, Bethesda, MD 20892, USA
| | - Debadatta Dash
- Human Cortical Physiology and Neurorehabilitation Section, NINDS, NIH, Bethesda, MD 20892, USA
| | | | - William Hayward
- Human Cortical Physiology and Neurorehabilitation Section, NINDS, NIH, Bethesda, MD 20892, USA
| | - Marlene Bönstrup
- Department of Neurology, University of Leipzig Medical Center, 04103 Leipzig, Germany
| | - Ethan R Buch
- Human Cortical Physiology and Neurorehabilitation Section, NINDS, NIH, Bethesda, MD 20892, USA
| | - Leonardo G Cohen
- Human Cortical Physiology and Neurorehabilitation Section, NINDS, NIH, Bethesda, MD 20892, USA.
| |
Collapse
|
14
|
Ehrlich SK, Dean-Leon E, Tacca N, Armleder S, Dimova-Edeleva V, Cheng G. Human-robot collaborative task planning using anticipatory brain responses. PLoS One 2023; 18:e0287958. [PMID: 37432954 DOI: 10.1371/journal.pone.0287958] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/06/2021] [Accepted: 06/19/2023] [Indexed: 07/13/2023] Open
Abstract
Human-robot interaction (HRI) describes scenarios in which both human and robot work as partners, sharing the same environment or complementing each other on a joint task. HRI is characterized by the need for high adaptability and flexibility of robotic systems toward their human interaction partners. One of the major challenges in HRI is task planning with dynamic subtask assignment, which is particularly challenging when subtask choices of the human are not readily accessible by the robot. In the present work, we explore the feasibility of using electroencephalogram (EEG) based neuro-cognitive measures for online robot learning of dynamic subtask assignment. To this end, we demonstrate in an experimental human subject study, featuring a joint HRI task with a UR10 robotic manipulator, the presence of EEG measures indicative of a human partner anticipating a takeover situation from human to robot or vice-versa. The present work further proposes a reinforcement learning based algorithm employing these measures as a neuronal feedback signal from the human to the robot for dynamic learning of subtask-assignment. The efficacy of this algorithm is validated in a simulation-based study. The simulation results reveal that even with relatively low decoding accuracies, successful robot learning of subtask-assignment is feasible, with around 80% choice accuracy among four subtasks within 17 minutes of collaboration. The simulation results further reveal that scalability to more subtasks is feasible and mainly accompanied with longer robot learning times. These findings demonstrate the usability of EEG-based neuro-cognitive measures to mediate the complex and largely unsolved problem of human-robot collaborative task planning.
Collapse
Affiliation(s)
- Stefan K Ehrlich
- Chair for Cognitive Systems, Department of Electrical Engineering, TUM School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
| | - Emmanuel Dean-Leon
- Department of Electrical Engineering, Automation, Chalmers University of Technology, Göteborg, Sweden
| | - Nicholas Tacca
- Battelle Memorial Institute, Columbus, OH, United States of America
| | - Simon Armleder
- Chair for Cognitive Systems, Department of Electrical Engineering, TUM School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
| | - Viktorija Dimova-Edeleva
- MIRMI - Munich Institute of Robotics and Machine Intelligence, formerly MSRM, Technical University of Munich, Munich, Germany
| | - Gordon Cheng
- Chair for Cognitive Systems, Department of Electrical Engineering, TUM School of Computation, Information and Technology, Technical University of Munich, Munich, Germany
- Center of Competence NeuroEngineering, Technical University of Munich, München, Germany
| |
Collapse
|
15
|
Yasemin M, Cruz A, Nunes UJ, Pires G. Single trial detection of error-related potentials in brain-machine interfaces: a survey and comparison of methods. J Neural Eng 2023; 20. [PMID: 36595316 DOI: 10.1088/1741-2552/acabe9] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2022] [Accepted: 12/15/2022] [Indexed: 12/23/2022]
Abstract
Objective.Error-related potential (ErrP) is a potential elicited in the brain when humans perceive an error. ErrPs have been researched in a variety of contexts, such as to increase the reliability of brain-computer interfaces (BCIs), increase the naturalness of human-machine interaction systems, teach systems, as well as study clinical conditions. Still, there is a significant challenge in detecting ErrP from a single trial, which may hamper its effective use. The literature presents ErrP detection accuracies quite variable across studies, which raises the question of whether this variability depends more on classification pipelines or on the quality of elicited ErrPs (mostly directly related to the underlying paradigms).Approach.With this purpose, 11 datasets have been used to compare several classification pipelines which were selected according to the studies that reported online performance above 75%. We also analyze the effects of different steps of the pipelines, such as resampling, window selection, augmentation, feature extraction, and classification.Main results.From our analysis, we have found that shrinkage-regularized linear discriminant analysis is the most robust method for classification, and for feature extraction, using Fisher criterion beamformer spatial features and overlapped window averages result in better classification performance. The overall experimental results suggest that classification accuracy is highly dependent on user tasks in BCI experiments and on signal quality (in terms of ErrP morphology, signal-to-noise ratio (SNR), and discrimination).Significance.This study contributes to the BCI research field by responding to the need for a guideline that can direct researchers in designing ErrP-based BCI tasks by accelerating the design steps.
Collapse
Affiliation(s)
- Mine Yasemin
- Institute of Systems and Robotics (ISR-UC), University of Coimbra, Coimbra, Portugal.,Department of Electrical and Computer Engineering, University of Coimbra, Coimbra, Portugal
| | - Aniana Cruz
- Institute of Systems and Robotics (ISR-UC), University of Coimbra, Coimbra, Portugal
| | - Urbano J Nunes
- Institute of Systems and Robotics (ISR-UC), University of Coimbra, Coimbra, Portugal.,Department of Electrical and Computer Engineering, University of Coimbra, Coimbra, Portugal
| | - Gabriel Pires
- Institute of Systems and Robotics (ISR-UC), University of Coimbra, Coimbra, Portugal.,Engineering Department, Polytechnic Institute of Tomar, Tomar, Portugal
| |
Collapse
|
16
|
Pires G, Cruz A, Jesus D, Yasemin M, Nunes UJ, Sousa T, Castelo-Branco M. A new error-monitoring brain-computer interface based on reinforcement learning for people with autism spectrum disorders. J Neural Eng 2022; 19. [PMID: 36541535 DOI: 10.1088/1741-2552/aca798] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/26/2022] [Accepted: 11/30/2022] [Indexed: 12/03/2022]
Abstract
Objective.Brain-computer interfaces (BCIs) are emerging as promising cognitive training tools in neurodevelopmental disorders, as they combine the advantages of traditional computerized interventions with real-time tailored feedback. We propose a gamified BCI based on non-volitional neurofeedback for cognitive training, aiming at reaching a neurorehabilitation tool for application in autism spectrum disorders (ASDs).Approach.The BCI consists of an emotional facial expression paradigm controlled by an intelligent agent that makes correct and wrong actions, while the user observes and judges the agent's actions. The agent learns through reinforcement learning (RL) an optimal strategy if the participant generates error-related potentials (ErrPs) upon incorrect agent actions. We hypothesize that this training approach will allow not only the agent to learn but also the BCI user, by participating through implicit error scrutiny in the process of learning through operant conditioning, making it of particular interest for disorders where error monitoring processes are altered/compromised such as in ASD. In this paper, the main goal is to validate the whole methodological BCI approach and assess whether it is feasible enough to move on to clinical experiments. A control group of ten neurotypical participants and one participant with ASD tested the proposed BCI approach.Main results.We achieved an online balanced-accuracy in ErrPs detection of 81.6% and 77.1%, respectively for two different game modes. Additionally, all participants achieved an optimal RL strategy for the agent at least in one of the test sessions.Significance.The ErrP classification results and the possibility of successfully achieving an optimal learning strategy, show the feasibility of the proposed methodology, which allows to move towards clinical experimentation with ASD participants to assess the effectiveness of the approach as hypothesized.
Collapse
Affiliation(s)
- Gabriel Pires
- Institute of Systems and Robotics of the University of Coimbra, Coimbra, Portugal.,Engineering Department, Polytechnic Institute of Tomar, Tomar, Portugal
| | - Aniana Cruz
- Institute of Systems and Robotics of the University of Coimbra, Coimbra, Portugal
| | - Diogo Jesus
- Institute of Systems and Robotics of the University of Coimbra, Coimbra, Portugal
| | - Mine Yasemin
- Institute of Systems and Robotics of the University of Coimbra, Coimbra, Portugal
| | - Urbano J Nunes
- Institute of Systems and Robotics of the University of Coimbra, Coimbra, Portugal.,Department of Electrical and Computer Engineering, University of Coimbra, Coimbra, Portugal
| | - Teresa Sousa
- Coimbra Institute for Biomedical Imaging and Translational Research of the University of Coimbra, Coimbra, Portugal
| | - Miguel Castelo-Branco
- Coimbra Institute for Biomedical Imaging and Translational Research of the University of Coimbra, Coimbra, Portugal.,Faculty of Medicine, University of Coimbra, Coimbra, Portugal
| |
Collapse
|
17
|
Wang X, Chen HT, Lin CT. Error-related potential-based shared autonomy via deep recurrent reinforcement learning. J Neural Eng 2022; 19. [PMID: 36541532 DOI: 10.1088/1741-2552/aca4fb] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2022] [Accepted: 11/22/2022] [Indexed: 11/23/2022]
Abstract
Objective.Error-related potential (ErrP)-based brain-computer interfaces (BCIs) have received a considerable amount of attention in the human-robot interaction community. In contrast to traditional BCI, which requires continuous and explicit commands from an operator, ErrP-based BCI leverages the ErrP, which is evoked when an operator observes unexpected behaviours from the robot counterpart. This paper proposes a novel shared autonomy model for ErrP-based human-robot interaction.Approach.We incorporate ErrP information provided by a BCI as useful observations for an agent and formulate the shared autonomy problem as a partially observable Markov decision process. A recurrent neural network-based actor-critic model is used to address the uncertainty in the ErrP signal. We evaluate the proposed framework in a simulated human-in-the-loop robot navigation task with both simulated users and real users.Main results.The results show that the proposed ErrP-based shared autonomy model enables an autonomous robot to complete navigation tasks more efficiently. In a simulation with 70% ErrP accuracy, agents completed the task 14.1% faster than in the no ErrP condition, while with real users, agents completed the navigation task 14.9% faster.Significance.The evaluation results confirmed that the shared autonomy via deep recurrent reinforcement learning is an effective way to deal with uncertain human feedback in a complex human-robot interaction task.
Collapse
Affiliation(s)
- Xiaofei Wang
- Computational Intelligence and Brain Computer Interface Lab, School of Computer Science, University of Technology, Sydney, NSW 2007, Australia
| | - Hsiang-Ting Chen
- School of Computer Science, University of Adelaide, Adelaide, SA 5005, Australia
| | - Chin-Teng Lin
- Computational Intelligence and Brain Computer Interface Lab, School of Computer Science, University of Technology, Sydney, NSW 2007, Australia
| |
Collapse
|
18
|
Hsieh JC, Alawieh H, Li Y, Iwane F, Zhao L, Anderson R, Abdullah S, Kevin Tang KW, Wang W, Pyatnitskiy I, Jia Y, Millán JDR, Wang H. A highly stable electrode with low electrode-skin impedance for wearable brain-computer interface. Biosens Bioelectron 2022; 218:114756. [DOI: 10.1016/j.bios.2022.114756] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2022] [Revised: 09/19/2022] [Accepted: 09/23/2022] [Indexed: 11/17/2022]
|
19
|
Xavier Fidêncio A, Klaes C, Iossifidis I. Error-Related Potentials in Reinforcement Learning-Based Brain-Machine Interfaces. Front Hum Neurosci 2022; 16:806517. [PMID: 35814961 PMCID: PMC9263570 DOI: 10.3389/fnhum.2022.806517] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2021] [Accepted: 05/30/2022] [Indexed: 11/13/2022] Open
Abstract
The human brain has been an object of extensive investigation in different fields. While several studies have focused on understanding the neural correlates of error processing, advances in brain-machine interface systems using non-invasive techniques further enabled the use of the measured signals in different applications. The possibility of detecting these error-related potentials (ErrPs) under different experimental setups on a single-trial basis has further increased interest in their integration in closed-loop settings to improve system performance, for example, by performing error correction. Fewer works have, however, aimed at reducing future mistakes or learning. We present a review focused on the current literature using non-invasive systems that have combined the ErrPs information specifically in a reinforcement learning framework to go beyond error correction and have used these signals for learning.
Collapse
Affiliation(s)
- Aline Xavier Fidêncio
- Robotics and BCI Laboratory, Institute of Computer Science, Ruhr West University of Applied Sciences, Mülheim an der Ruhr, Germany
- KlaesLab, Department of Neurosurgery, University Hospital Knappschaftskrankenhaus Bochum GmbH, Bochum, Germany
- Faculty of Electrical Engineering and Information Technology, Ruhr-University Bochum, Bochum, Germany
- *Correspondence: Aline Xavier Fidêncio
| | - Christian Klaes
- KlaesLab, Department of Neurosurgery, University Hospital Knappschaftskrankenhaus Bochum GmbH, Bochum, Germany
| | - Ioannis Iossifidis
- Robotics and BCI Laboratory, Institute of Computer Science, Ruhr West University of Applied Sciences, Mülheim an der Ruhr, Germany
| |
Collapse
|
20
|
Parashiva PK, Vinod A. Improving direction decoding accuracy during online motor imagery based brain-computer interface using error-related potentials. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103515] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
21
|
Parashiva PK, Vinod AP. Single-trial detection of EEG error-related potentials in serial visual presentation paradigm. Biomed Phys Eng Express 2021; 8. [PMID: 34891146 DOI: 10.1088/2057-1976/ac4200] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/04/2021] [Accepted: 12/10/2021] [Indexed: 11/11/2022]
Abstract
When the outcome of an event is not the same as expected, the cognitive state that monitors performance elicits a time-locked brain response termed as Error-Related Potential (ErrP).Objective-In the existing work, ErrP is not recorded when there is a disassociation between an object and its description. The objective of this work is to propose a Serial Visual Presentation (SVP) experimental paradigm to record ErrP when an image and its label are disassociated. Additionally, this work aims to propose a novel method for detecting ErrP on a single-trial basis.Method-The method followed in this work includes designing of SVP paradigm in which labeled images from six categories (bike, car, flower, fruit, cat, and dog) are presented serially. In this work, a text (visual) or an audio clip describing the image in one word is presented as the label. Further, the ErrP is detected on a single-trial basis using novel electrode-averaged features.Results -The ErrP data recorded from 11 subjects' have consistent characteristics compared to existing ErrP literature. Detection of ErrP on a single-trial basis is carried out using a novel feature extraction method on two type labeling types separately. The best average classification accuracy achieved is69.09±4.70%and63.33±4.56%for the audio and visual type of labeling the image, respectively. The proposed feature extraction method achieved higher classification accuracy when compared with two existing feature extraction methods.Significance -The significance of this work is that it can be used as a Brain-Computer Interface (BCI) system for quantitative evaluation and treatment of mild cognitive impairment. This work can also find non-clinical BCI applications such as image annotation.
Collapse
Affiliation(s)
- Praveen K Parashiva
- Department of Electrical Engineering, Indian Institute of Technology Palakkad, India
| | - A P Vinod
- Department of Electrical Engineering, Indian Institute of Technology Palakkad, India.,Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Hong Kong, People's Republic of China
| |
Collapse
|
22
|
Customizing skills for assistive robotic manipulators, an inverse reinforcement learning approach with error-related potentials. Commun Biol 2021; 4:1406. [PMID: 34916587 PMCID: PMC8677775 DOI: 10.1038/s42003-021-02891-8] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/10/2020] [Accepted: 11/10/2021] [Indexed: 11/09/2022] Open
Abstract
Robotic assistance via motorized robotic arm manipulators can be of valuable assistance to individuals with upper-limb motor disabilities. Brain-computer interfaces (BCI) offer an intuitive means to control such assistive robotic manipulators. However, BCI performance may vary due to the non-stationary nature of the electroencephalogram (EEG) signals. It, hence, cannot be used safely for controlling tasks where errors may be detrimental to the user. Avoiding obstacles is one such task. As there exist many techniques to avoid obstacles in robotics, we propose to give the control to the robot to avoid obstacles and to leave to the user the choice of the robot behavior to do so a matter of personal preference as some users may be more daring while others more careful. We enable the users to train the robot controller to adapt its way to approach obstacles relying on BCI that detects error-related potentials (ErrP), indicative of the user’s error expectation of the robot’s current strategy to meet their preferences. Gaussian process-based inverse reinforcement learning, in combination with the ErrP-BCI, infers the user’s preference and updates the obstacle avoidance controller so as to generate personalized robot trajectories. We validate the approach in experiments with thirteen able-bodied subjects using a robotic arm that picks up, places and avoids real-life objects. Results show that the algorithm can learn user’s preference and adapt the robot behavior rapidly using less than five demonstrations not necessarily optimal. Teaching an assistive robotic manipulator to move objects in a cluttered table requires demonstrations from expert operators, but what if the experts are individuals with motor disabilities? Batzianoulis et al. propose a learning approach which combines robot autonomy and a brain-computer interfacing that decodes whether the generated trajectories meet the user’s criteria, and show how their system enables the robot to learn individual user’s preferred behaviors using less than five demonstrations that are not necessarily optimal.
Collapse
|
23
|
Meyer SM, Rao Mangalore A, Ehrlich SK, Berberich N, Nassour J, Cheng G. A Comparative Pilot Study on ErrPs for Different Usage Conditions of an Exoskeleton with a Mobile EEG Device. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2021; 2021:6203-6206. [PMID: 34892532 DOI: 10.1109/embc46164.2021.9630465] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/14/2023]
Abstract
Exoskeletons and prosthetic devices controlled using brain-computer interfaces (BCIs) can be prone to errors due to inconsistent decoding. In recent years, it has been demonstrated that error-related potentials (ErrPs) can be used as a feedback signal in electroencephalography (EEG) based BCIs. However, modern BCIs often take large setup times and are physically restrictive, making them impractical for everyday use. In this paper, we use a mobile and easy-to-setup EEG device to investigate whether an erroneously functioning 1-DOF exoskeleton in different conditions, namely, visually observing and wearing the exoskeleton, elicits a brain response that can be classified. We develop a pipeline that can be applied to these two conditions and observe from our experiments that there is evidence for neural responses from electrodes near regions associated with ErrPs in an environment that resembles the real world. We found that these error-related responses can be classified as ErrPs with accuracies ranging from 60% to 71%, depending on the condition and the subject. Our pipeline could be further extended to detect and correct erroneous exoskeleton behavior in real-world settings.
Collapse
|
24
|
Barresi G, Marinelli A, Caserta G, de Zambotti M, Tessadori J, Angioletti L, Boccardo N, Freddolini M, Mazzanti D, Deshpande N, Frigo CA, Balconi M, Gruppioni E, Laffranchi M, De Michieli L. Exploring the Embodiment of a Virtual Hand in a Spatially Augmented Respiratory Biofeedback Setting. Front Neurorobot 2021; 15:683653. [PMID: 34557082 PMCID: PMC8454775 DOI: 10.3389/fnbot.2021.683653] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2021] [Accepted: 07/26/2021] [Indexed: 01/15/2023] Open
Abstract
Enhancing the embodiment of artificial limbs—the individuals' feeling that a virtual or robotic limb is integrated in their own body scheme—is an impactful strategy for improving prosthetic technology acceptance and human-machine interaction. Most studies so far focused on visuo-tactile strategies to empower the embodiment processes. However, novel approaches could emerge from self-regulation techniques able to change the psychophysiological conditions of an individual. Accordingly, this pilot study investigates the effects of a self-regulated breathing exercise on the processes of body ownership underlying the embodiment of a virtual right hand within a Spatially Augmented Respiratory Biofeedback (SARB) setting. This investigation also aims at evaluating the feasibility of the breathing exercise enabled by a low-cost SARB implementation designed for upcoming remote studies (a need emerged during the COVID-19 pandemic). Twenty-two subjects without impairments, and two transradial prosthesis users for a preparatory test, were asked (in each condition of a within-group design) to maintain a normal (about 14 breaths/min) or slow (about 6 breaths/min) respiratory rate to keep a static virtual right hand “visible” on a screen. Meanwhile, a computer-generated sphere moved from left to right toward the virtual hand during each trial (1 min) of 16. If the participant's breathing rate was within the target (slow or normal) range, a visuo-tactile event was triggered by the sphere passing under the virtual hand (the subjects observed it shaking while they perceived a vibratory feedback generated by a smartphone). Our results—mainly based on questionnaire scores and proprioceptive drift—highlight that the slow breathing condition induced higher embodiment than the normal one. This preliminary study reveals the feasibility and potential of a novel psychophysiological training strategy to enhance the embodiment of artificial limbs. Future studies are needed to further investigate mechanisms, efficacy and generalizability of the SARB techniques in training a bionic limb embodiment.
Collapse
Affiliation(s)
- Giacinto Barresi
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Andrea Marinelli
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy.,Department of Informatics, Bioengineering, Robotics, and Systems Engineering, Università degli Studi di Genova, Genoa, Italy
| | - Giulia Caserta
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy.,Movement Biomechanics and Motor Control Lab, Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| | | | - Jacopo Tessadori
- Visual Geometry and Modelling, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Laura Angioletti
- International Research Center for Cognitive Applied Neuroscience, Università Cattolica del Sacro Cuore, Milan, Italy.,Research Unit in Affective and Social Neuroscience, Department of Psychology, Università Cattolica del Sacro Cuore, Milan, Italy
| | - Nicolò Boccardo
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Marco Freddolini
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Dario Mazzanti
- Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Nikhil Deshpande
- Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Carlo Albino Frigo
- Movement Biomechanics and Motor Control Lab, Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| | - Michela Balconi
- International Research Center for Cognitive Applied Neuroscience, Università Cattolica del Sacro Cuore, Milan, Italy.,Research Unit in Affective and Social Neuroscience, Department of Psychology, Università Cattolica del Sacro Cuore, Milan, Italy
| | - Emanuele Gruppioni
- Centro Protesi INAIL, Istituto Nazionale per l'Assicurazione contro gli Infortuni sul Lavoro, Bologna, Italy
| | | | | |
Collapse
|
25
|
Min BK, Kim HS, Ko W, Ahn MH, Suk HI, Pantazis D, Knight RT. Electrophysiological Decoding of Spatial and Color Processing in Human Prefrontal Cortex. Neuroimage 2021; 237:118165. [PMID: 34000400 PMCID: PMC8344402 DOI: 10.1016/j.neuroimage.2021.118165] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/28/2021] [Revised: 04/30/2021] [Accepted: 05/11/2021] [Indexed: 11/16/2022] Open
Abstract
The prefrontal cortex (PFC) plays a pivotal role in goal-directed cognition, yet its representational code remains an open problem with decoding techniques ineffective in disentangling task-relevant variables from PFC. Here we applied regularized linear discriminant analysis to human scalp EEG data and were able to distinguish a mental-rotation task versus a color-perception task with 87% decoding accuracy. Dorsal and ventral areas in lateral PFC provided the dominant features dissociating the two tasks. Our findings show that EEG can reliably decode two independent task states from PFC and emphasize the PFC dorsal/ventral functional specificity in processing the where rotation task versus the what color task.
Collapse
Affiliation(s)
- Byoung-Kyong Min
- Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, Korea; Department of Artificial Intelligence, Korea University, Seoul 02841, Korea.
| | - Hyun-Seok Kim
- Biomedical Engineering Research Center, Asan Institute of Life Science, Asan Medical Center, Seoul 05505, Korea
| | - Wonjun Ko
- Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, Korea
| | - Min-Hee Ahn
- Laboratory of Brain and Cognitive Science for Convergence Medicine, College of Medicine, Hallym University, Anyang 14068, Korea
| | - Heung-Il Suk
- Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, Korea; Department of Artificial Intelligence, Korea University, Seoul 02841, Korea
| | - Dimitrios Pantazis
- McGovern Institute for Brain Research, Massachusetts Institute of Technology, Cambridge, MA 02139, USA
| | - Robert T Knight
- Department of Psychology, Helen Wills Neuroscience Institute, UC Berkeley, Berkeley, CA 94720, USA
| |
Collapse
|
26
|
Czeszumski A, Gert AL, Keshava A, Ghadirzadeh A, Kalthoff T, Ehinger BV, Tiessen M, Björkman M, Kragic D, König P. Coordinating With a Robot Partner Affects Neural Processing Related to Action Monitoring. Front Neurorobot 2021; 15:686010. [PMID: 34456705 PMCID: PMC8386170 DOI: 10.3389/fnbot.2021.686010] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/26/2021] [Accepted: 06/30/2021] [Indexed: 11/24/2022] Open
Abstract
Robots start to play a role in our social landscape, and they are progressively becoming responsive, both physically and socially. It begs the question of how humans react to and interact with robots in a coordinated manner and what the neural underpinnings of such behavior are. This exploratory study aims to understand the differences in human-human and human-robot interactions at a behavioral level and from a neurophysiological perspective. For this purpose, we adapted a collaborative dynamical paradigm from the literature. We asked 12 participants to hold two corners of a tablet while collaboratively guiding a ball around a circular track either with another participant or a robot. In irregular intervals, the ball was perturbed outward creating an artificial error in the behavior, which required corrective measures to return to the circular track again. Concurrently, we recorded electroencephalography (EEG). In the behavioral data, we found an increased velocity and positional error of the ball from the track in the human-human condition vs. human-robot condition. For the EEG data, we computed event-related potentials. We found a significant difference between human and robot partners driven by significant clusters at fronto-central electrodes. The amplitudes were stronger with a robot partner, suggesting a different neural processing. All in all, our exploratory study suggests that coordinating with robots affects action monitoring related processing. In the investigated paradigm, human participants treat errors during human-robot interaction differently from those made during interactions with other humans. These results can improve communication between humans and robot with the use of neural activity in real-time.
Collapse
Affiliation(s)
- Artur Czeszumski
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
| | - Anna L. Gert
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
| | - Ashima Keshava
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
| | - Ali Ghadirzadeh
- Robotics, Perception and Learning, School of Electrical Engineering and Computer Science, Kungliga Tekniska Högskolan Royal Institute of Technology, Stockholm, Sweden
| | - Tilman Kalthoff
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
| | - Benedikt V. Ehinger
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
- Stuttgart Center for Simulation Science, University of Stuttgart, Stuttgart, Germany
| | - Max Tiessen
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
| | - Mårten Björkman
- Robotics, Perception and Learning, School of Electrical Engineering and Computer Science, Kungliga Tekniska Högskolan Royal Institute of Technology, Stockholm, Sweden
| | - Danica Kragic
- Robotics, Perception and Learning, School of Electrical Engineering and Computer Science, Kungliga Tekniska Högskolan Royal Institute of Technology, Stockholm, Sweden
| | - Peter König
- Institute of Cognitive Science, Universität Osnabrück, Osnabrück, Germany
- Institut für Neurophysiologie und Pathophysiologie, Universitätsklinikum Hamburg-Eppendorf, Hamburg, Germany
| |
Collapse
|
27
|
Cao L, Li G, Xu Y, Zhang H, Shu X, Zhang D. A brain-actuated robotic arm system using non-invasive hybrid brain-computer interface and shared control strategy. J Neural Eng 2021; 18. [PMID: 33862607 DOI: 10.1088/1741-2552/abf8cb] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/26/2020] [Accepted: 04/16/2021] [Indexed: 01/20/2023]
Abstract
Objective.The electroencephalography (EEG)-based brain-computer interfaces (BCIs) have been used in the control of robotic arms. The performance of non-invasive BCIs may not be satisfactory due to the poor quality of EEG signals, so the shared control strategies were tried as an alternative solution. However, most of the existing shared control methods set the arbitration rules manually, which highly depended on the specific tasks and developer's experience. In this study, we proposed a novel shared control model that automatically optimized the control commands in a dynamical way based on the context in real-time control. Besides, we employed the hybrid BCI to better allocate commands with multiple functions. The system allowed non-invasive BCI users to manipulate a robotic arm moving in a three-dimensional (3D) space and complete a pick-place task of multiple objects.Approach.Taking the scene information obtained by computer vision as a knowledge base, a machine agent was designed to infer the user's intention and generate automatic commands. Based on the inference confidence and user's characteristic, the proposed shared control model fused the machine autonomy and human intention dynamically for robotic arm motion optimization during the online control. In addition, we introduced a hybrid BCI scheme that applied steady-state visual evoked potentials and motor imagery to the divided primary and secondary BCI interfaces to better allocate the BCI resources (e.g. decoding computing power, screen occupation) and realize the multi-dimensional control of the robotic arm.Main results.Eleven subjects participated in the online experiments of picking and placing five objects that scattered at different positions in a 3D workspace. The results showed that most of the subjects could control the robotic arm to complete accurate and robust picking task with an average success rate of approximately 85% under the shared control strategy, while the average success rate of placing task controlled by pure BCI was 50% approximately.Significance.In this paper, we proposed a novel shared controller for motion automatic optimization, together with a hybrid BCI control scheme that allocated paradigms according to the importance of commands to realize multi-dimensional and effective control of a robotic arm. Our study indicated that the shared control strategy with hybrid BCI could greatly improve the performance of the brain-actuated robotic arm system.
Collapse
Affiliation(s)
- Linfeng Cao
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Guangye Li
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Yang Xu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Heng Zhang
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Xiaokang Shu
- State Key Laboratory of Mechanical Systems and Vibrations, Institute of Robotics, Shanghai Jiao Tong University, Shanghai, People's Republic of China
| | - Dingguo Zhang
- Department of Electronic and Electrical Engineering, University of Bath, Bath, United Kingdom
| |
Collapse
|
28
|
Iwane F, Iturrate I, Chavarriaga R, Millán JDR. Invariability of EEG error-related potentials during continuous feedback protocols elicited by erroneous actions at predicted or unpredicted states. J Neural Eng 2021; 18. [PMID: 33882461 DOI: 10.1088/1741-2552/abfa70] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2020] [Accepted: 04/21/2021] [Indexed: 11/11/2022]
Abstract
Objective.When humans perceive an erroneous action, an EEG error-related potential (ErrP) is elicited as a neural response. ErrPs have been largely investigated in discrete feedback protocols, where actions are executed at discrete steps, to enable seamless brain-computer interaction. However, there are only a few studies that investigate ErrPs in continuous feedback protocols. The objective of the present study is to better understand the differences between two types of ErrPs elicited during continuous feedback protocols, where errors may occur either at predicted or unpredicted states. We hypothesize that ErrPs of the unpredicted state is associated with longer latency as it requires higher cognitive workload to evaluate actions compared to the predicted states.Approach.Participants monitored the trajectory of an autonomous cursor that occasionally made erroneous actions on its way to the target in two conditions, namely, predicted or unpredicted states. After characterizing the ErrP waveform elicited by erroneous actions in the two conditions, we performed single-trial decoding of ErrPs in both synchronous (i.e. time-locked to the onset of the erroneous action) and asynchronous manner. Furthermore, we explored the possibility to transfer decoders built with data of one of the conditions to the other condition.Main results.As hypothesized, erroneous actions at unpredicted states gave rise to ErrPs with higher latency than erroneous actions at predicted states, a correlate of higher cognitive effort in the former condition. Moreover, ErrP decoders trained in a given condition successfully transferred to the other condition with a slight loss of classification performance. This was the case for synchronous as well as asynchronous ErrP decoding, showing the invariability of ErrPs across conditions.Significance.These results advance the characterization of ErrPs during continuous feedback protocols, enlarging the potential use of ErrPs during natural operation of brain-controlled devices as it is not necessary to have different decoders for each kind of erroneous conditions.
Collapse
Affiliation(s)
- Fumiaki Iwane
- Learning Algorithms and Systems Laboratory (LASA) , École Polytechnique Féderale de Lausanne (EPFL), 1015 Lausanne, Switzerland.,Department of Electrical and Computer Engineering , The University of Texas at Austin, Austin, TX 78712, United States of America
| | - Iñaki Iturrate
- École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,Amazon , Barcelona, Spain
| | - Ricardo Chavarriaga
- École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,ZHAW Datalab , Zurich University of Applied Sciences, Winterthur, Switzerland
| | - José Del R Millán
- Department of Electrical and Computer Engineering , The University of Texas at Austin, Austin, TX 78712, United States of America.,École Polytechnique Féderale de Lausanne (EPFL), Campus Biotech , 1202 Genève, Switzerland.,Department of Neurology , The University of Texas at Austin, Austin, TX 78712, United States of America
| |
Collapse
|
29
|
Parashiva PK, Vinod A. Single-trial detection of EEG error-related potentials using modified power-law transformation. Biomed Signal Process Control 2021. [DOI: 10.1016/j.bspc.2021.102563] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2022]
|
30
|
Levi-Aharoni H, Tishby N. The value-complexity trade-off for reinforcement learning based brain-computer interfaces. J Neural Eng 2021; 17:066011. [PMID: 33586668 DOI: 10.1088/1741-2552/abc8d8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
Abstract
OBJECTIVE One of the recent developments in the field of brain-computer interfaces (BCI) is the reinforcement learning (RL) based BCI paradigm, which uses neural error responses as the reward feedback on the agent's action. While having several advantages over motor imagery based BCI, the reliability of RL-BCI is critically dependent on the decoding accuracy of noisy neural error signals. A principled method is needed to optimally handle this inherent noise under general conditions. APPROACH By determining a trade-off between the expected value and the informational cost of policies, the info-RL (IRL) algorithm provides optimal low-complexity policies, which are robust under noisy reward conditions and achieve the maximal obtainable value. In this work we utilize the IRL algorithm to characterize the maximal obtainable value under different noise levels, which in turn is used to extract the optimal robust policy for each noise level. MAIN RESULTS Our simulation results of a setting with Gaussian noise show that the complexity level of the optimal policy is dependent on the reward magnitude but not on the reward variance, whereas the variance determines whether a lower complexity solution is favorable or not. We show how this analysis can be utilized to select optimal robust policies for an RL-BCI and demonstrate its use on EEG data. SIGNIFICANCE We propose here a principled method to determine the optimal policy complexity of an RL problem with a noisy reward, which we argue is particularly useful for RL-based BCI paradigms. This framework may be used to minimize initial training time and allow for a more dynamic and robust shared control between the agent and the operator under different conditions.
Collapse
Affiliation(s)
- Hadar Levi-Aharoni
- The Edmond and Lilly Safra Center for Brain Sciences, Hebrew University of Jerusalem, Jerusalem, Israel
| | | |
Collapse
|
31
|
Kim SK, Kirchner EA, Schloßmüller L, Kirchner F. Errors in Human-Robot Interactions and Their Effects on Robot Learning. Front Robot AI 2021; 7:558531. [PMID: 33501322 PMCID: PMC7805941 DOI: 10.3389/frobt.2020.558531] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2020] [Accepted: 08/19/2020] [Indexed: 11/13/2022] Open
Abstract
During human-robot interaction, errors will occur. Hence, understanding the effects of interaction errors and especially the effect of prior knowledge on robot learning performance is relevant to develop appropriate approaches for learning under natural interaction conditions, since future robots will continue to learn based on what they have already learned. In this study, we investigated interaction errors that occurred under two learning conditions, i.e., in the case that the robot learned without prior knowledge (cold-start learning) and in the case that the robot had prior knowledge (warm-start learning). In our human-robot interaction scenario, the robot learns to assign the correct action to a current human intention (gesture). Gestures were not predefined but the robot had to learn their meaning. We used a contextual-bandit approach to maximize the expected payoff by updating (a) the current human intention (gesture) and (b) the current human intrinsic feedback after each action selection of the robot. As an intrinsic evaluation of the robot behavior we used the error-related potential (ErrP) in the human electroencephalogram as reinforcement signal. Either gesture errors (human intentions) can be misinterpreted by incorrectly captured gestures or errors in the ErrP classification (human feedback) can occur. We investigated these two types of interaction errors and their effects on the learning process. Our results show that learning and its online adaptation was successful under both learning conditions (except for one subject in cold-start learning). Furthermore, warm-start learning achieved faster convergence, while cold-start learning was less affected by online changes in the current context.
Collapse
Affiliation(s)
- Su Kyoung Kim
- Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI GmbH), Bremen, Germany
| | - Elsa Andrea Kirchner
- Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI GmbH), Bremen, Germany.,Research Group Robotics, University of Bremen, Bremen, Germany
| | | | - Frank Kirchner
- Robotics Innovation Center, German Research Center for Artificial Intelligence (DFKI GmbH), Bremen, Germany.,Research Group Robotics, University of Bremen, Bremen, Germany
| |
Collapse
|
32
|
Cheng G, Ehrlich SK, Lebedev M, Nicolelis MAL. Neuroengineering challenges of fusing robotics and neuroscience. Sci Robot 2020; 5:5/49/eabd1911. [DOI: 10.1126/scirobotics.abd1911] [Citation(s) in RCA: 16] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/05/2020] [Revised: 10/01/2020] [Accepted: 11/16/2020] [Indexed: 11/02/2022]
Affiliation(s)
- Gordon Cheng
- Chair for Cognitive Systems, Department of Electrical and Computer Engineering, Technical University of Munich, Arcisstrasse 21, 80333 München, Germany
- Center of Competence NeuroEngineering, Technical University of Munich, Arcisstrasse 21, 80333 München, Germany
| | - Stefan K. Ehrlich
- Chair for Cognitive Systems, Department of Electrical and Computer Engineering, Technical University of Munich, Arcisstrasse 21, 80333 München, Germany
| | - Mikhail Lebedev
- Center for Bioelectric Interfaces of the Institute for Cognitive Neuroscience, National Research University Higher School of Economics, Myasnitskaya 13, building 4, 101000 Moscow, Russia
- I.M. Sechenov First Moscow State Medical University, 2-4 Bolshaya Pirogovskaya Street, 119991 Moscow, Russia
- Department of Neurobiology, Duke University Center for Neuroengineering, Duke University, Durham, NC 27710, USA
| | - Miguel A. L. Nicolelis
- Department of Neurobiology, Duke University Center for Neuroengineering, Duke University, Durham, NC 27710, USA
- Departments of Biomedical Engineering and Psychology and Neuroscience, Duke University, Durham, NC 27710, USA
- Edmund and Lily Safra International Institute of Neurosciences, Natal 59066060, Brazil
| |
Collapse
|
33
|
Alimardani M, Hiraki K. Passive Brain-Computer Interfaces for Enhanced Human-Robot Interaction. Front Robot AI 2020; 7:125. [PMID: 33501291 PMCID: PMC7805996 DOI: 10.3389/frobt.2020.00125] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/05/2020] [Accepted: 08/05/2020] [Indexed: 11/13/2022] Open
Abstract
Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.
Collapse
Affiliation(s)
- Maryam Alimardani
- Department of Cognitive Science and Artificial Intelligence, School of Humanities and Digital Sciences, Tilburg University, Tilburg, Netherlands
| | - Kazuo Hiraki
- Department of General Systems Studies, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
34
|
Xu R, Wang Y, Shi X, Wang N, Ming D. The Effect of Static and Dynamic Visual Stimulations on Error Detection Based on Error-Evoked Brain Responses. SENSORS 2020; 20:s20164475. [PMID: 32785187 PMCID: PMC7472474 DOI: 10.3390/s20164475] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/15/2020] [Revised: 07/13/2020] [Accepted: 07/30/2020] [Indexed: 12/03/2022]
Abstract
Error-related potentials (ErrPs) have provided technical support for the brain-computer interface. However, different visual stimulations may affect the ErrPs, and furthermore, affect the error recognition based on ErrPs. Therefore, the study aimed to investigate how people respond to different visual stimulations (static and dynamic) and find the best time window for different stimulation. Nineteen participants were recruited in the ErrPs-based tasks with static and dynamic visual stimulations. Five ErrPs were statistically compared, and the classification accuracies were obtained through linear discriminant analysis (LDA) with nine different time windows. The results showed that the P3, N6, and P8 with correctness were significantly different from those with error in both stimulations, while N1 only existed in static. The differences between dynamic and static errors existed in N1 and P2. The highest accuracy was obtained in the time window related to N1, P3, N6, and P8 for the static condition, and in the time window related to P3, N6, and P8 for the dynamic. In conclusion, the early components of ErrPs may be affected by stimulation modes, and the late components are more sensitive to errors. The error recognition with static stimulation requires information from the entire epoch, while the late windows should be focused more within the dynamic case.
Collapse
Affiliation(s)
- Rui Xu
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China; (R.X.); (Y.W.)
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, China; (X.S.); (N.W.)
| | - Yaoyao Wang
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China; (R.X.); (Y.W.)
| | - Xianle Shi
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, China; (X.S.); (N.W.)
| | - Ningning Wang
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, China; (X.S.); (N.W.)
| | - Dong Ming
- Academy of Medical Engineering and Translational Medicine, Tianjin University, Tianjin 300072, China; (R.X.); (Y.W.)
- Department of Biomedical Engineering, College of Precision Instruments and Optoelectronics Engineering, Tianjin University, Tianjin 300072, China; (X.S.); (N.W.)
- Correspondence:
| |
Collapse
|
35
|
Plug-and-play supervisory control using muscle and brain signals for real-time gesture and error detection. Auton Robots 2020. [DOI: 10.1007/s10514-020-09916-x] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/26/2022]
Abstract
AbstractEffective human supervision of robots can be key for ensuring correct robot operation in a variety of potentially safety-critical scenarios. This paper takes a step towards fast and reliable human intervention in supervisory control tasks by combining two streams of human biosignals: muscle and brain activity acquired via EMG and EEG, respectively. It presents continuous classification of left and right hand-gestures using muscle signals, time-locked classification of error-related potentials using brain signals (unconsciously produced when observing an error), and a framework that combines these pipelines to detect and correct robot mistakes during multiple-choice tasks. The resulting hybrid system is evaluated in a “plug-and-play” fashion with 7 untrained subjects supervising an autonomous robot performing a target selection task. Offline analysis further explores the EMG classification performance, and investigates methods to select subsets of training data that may facilitate generalizable plug-and-play classifiers.
Collapse
|
36
|
Schonleitner FM, Otter L, Ehrlich SK, Cheng G. Calibration-Free Error-Related Potential Decoding With Adaptive Subject-Independent Models: A Comparative Study. ACTA ACUST UNITED AC 2020. [DOI: 10.1109/tmrb.2020.3012436] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
37
|
Wirth C, Toth J, Arvaneh M. Four-Way Classification of EEG Responses To Virtual Robot Navigation. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:3050-3053. [PMID: 33018648 DOI: 10.1109/embc44109.2020.9176230] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/11/2023]
Abstract
Studies have shown the possibility of using brain signals that are automatically generated while observing a navigation task as feedback for semi-autonomous control of a robot. This allows the robot to learn quasi-optimal routes to intended targets. We have combined the subclassification of two different types of navigational errors, with the subclassification of two different types of correct navigational actions, to create a 4-way classification strategy, providing detailed information about the type of action the robot performed. We used a 2-stage stepwise linear discriminant analysis approach, and tested this using brain signals from 8 and 14 participants observing two robot navigation tasks. Classification results were significantly above the chance level, with mean overall accuracy of 44.3% and 36.0% for the two datasets. As a proof of concept, we have shown that it is possible to perform fine-grained, 4-way classification of robot navigational actions, based on the electroencephalogram responses of participants who only had to observe the task. This study provides the next step towards comprehensive implicit brain-machine communication, and towards an efficient semi-autonomous brain-computer interface.
Collapse
|
38
|
Jeong JH, Shim KH, Kim DJ, Lee SW. Brain-Controlled Robotic Arm System Based on Multi-Directional CNN-BiLSTM Network Using EEG Signals. IEEE Trans Neural Syst Rehabil Eng 2020; 28:1226-1238. [DOI: 10.1109/tnsre.2020.2981659] [Citation(s) in RCA: 83] [Impact Index Per Article: 20.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
39
|
Pereira M, Faivre N, Iturrate I, Wirthlin M, Serafini L, Martin S, Desvachez A, Blanke O, Van De Ville D, Millán JDR. Disentangling the origins of confidence in speeded perceptual judgments through multimodal imaging. Proc Natl Acad Sci U S A 2020; 117:8382-8390. [PMID: 32238562 PMCID: PMC7165419 DOI: 10.1073/pnas.1918335117] [Citation(s) in RCA: 29] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/14/2022] Open
Abstract
The human capacity to compute the likelihood that a decision is correct-known as metacognition-has proven difficult to study in isolation as it usually cooccurs with decision making. Here, we isolated postdecisional from decisional contributions to metacognition by analyzing neural correlates of confidence with multimodal imaging. Healthy volunteers reported their confidence in the accuracy of decisions they made or decisions they observed. We found better metacognitive performance for committed vs. observed decisions, indicating that committing to a decision may improve confidence. Relying on concurrent electroencephalography and hemodynamic recordings, we found a common correlate of confidence following committed and observed decisions in the inferior frontal gyrus and a dissociation in the anterior prefrontal cortex and anterior insula. We discuss these results in light of decisional and postdecisional accounts of confidence and propose a computational model of confidence in which metacognitive performance naturally improves when evidence accumulation is constrained upon committing a decision.
Collapse
Affiliation(s)
- Michael Pereira
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland;
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Faculty of Life Sciences, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Laboratoire de Psychologie et Neurocognition, CNRS UMR 5105, Université Grenoble Alpes, 38400 Saint-Martin-d'Hères, France
| | - Nathan Faivre
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Faculty of Life Sciences, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Laboratoire de Psychologie et Neurocognition, CNRS UMR 5105, Université Grenoble Alpes, 38400 Saint-Martin-d'Hères, France
| | - Iñaki Iturrate
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
| | - Marco Wirthlin
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Faculty of Life Sciences, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
| | - Luana Serafini
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Faculty of Life Sciences, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Department of Biomedical, Metabolic and Neural Sciences, University of Modena and Reggio Emilia, 41121 Modena, Italy
| | - Stéphanie Martin
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
| | - Arnaud Desvachez
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
| | - Olaf Blanke
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Laboratory of Cognitive Neuroscience, Brain Mind Institute, Faculty of Life Sciences, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Department of Neurology, University Hospital Geneva, 1205 Geneva, Switzerland
| | - Dimitri Van De Ville
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Medical Image Processing Lab, Institute of Bioengineering, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Department of Radiology and Medical Informatics, University of Geneva, 1205 Geneva, Switzerland
| | - José Del R Millán
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, 1202 Geneva, Switzerland
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX 78712
- Department of Neurology, The University of Texas at Austin, Austin, TX 78712
| |
Collapse
|
40
|
Iturrate I, Chavarriaga R, Millán JDR. General principles of machine learning for brain-computer interfacing. HANDBOOK OF CLINICAL NEUROLOGY 2020; 168:311-328. [PMID: 32164862 DOI: 10.1016/b978-0-444-63934-9.00023-8] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Indexed: 11/19/2022]
Abstract
Brain-computer interfaces (BCIs) are systems that translate brain activity patterns into commands that can be executed by an artificial device. This enables the possibility of controlling devices such as a prosthetic arm or exoskeleton, a wheelchair, typewriting applications, or games directly by modulating our brain activity. For this purpose, BCI systems rely on signal processing and machine learning algorithms to decode the brain activity. This chapter provides an overview of the main steps required to do such a process, including signal preprocessing, feature extraction and selection, and decoding. Given the large amount of possible methods that can be used for these processes, a comprehensive review of them is beyond the scope of this chapter, and it is focused instead on the general principles that should be taken into account, as well as discussing good practices on how these methods should be applied and evaluated for proper design of reliable BCI systems.
Collapse
Affiliation(s)
- Iñaki Iturrate
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Ricardo Chavarriaga
- Center for Neuroprosthetics, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland; Institute of Applied Information Technology (InIT), Zurich University of Applied Sciences ZHAW, Winterthur, Switzerland.
| | - José Del R Millán
- Department of Electrical and Computer Engineering, The University of Texas at Austin, Austin, TX, United States; Department of Neurology, The University of Texas at Austin, Austin, TX, United States
| |
Collapse
|
41
|
Wirth C, Toth J, Arvaneh M. "You Have Reached Your Destination": A Single Trial EEG Classification Study. Front Neurosci 2020; 14:66. [PMID: 32116513 PMCID: PMC7027274 DOI: 10.3389/fnins.2020.00066] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/03/2019] [Accepted: 01/16/2020] [Indexed: 01/24/2023] Open
Abstract
Studies have established that it is possible to differentiate between the brain's responses to observing correct and incorrect movements in navigation tasks. Furthermore, these classifications can be used as feedback for a learning-based BCI, to allow real or virtual robots to find quasi-optimal routes to a target. However, when navigating it is important not only to know we are moving in the right direction toward a target, but also to know when we have reached it. We asked participants to observe a virtual robot performing a 1-dimensional navigation task. We recorded EEG and then performed neurophysiological analysis on the responses to two classes of correct movements: those that moved closer to the target but did not reach it, and those that did reach the target. Further, we used a stepwise linear classifier on time-domain features to differentiate the classes on a single-trial basis. A second data set was also used to further test this single-trial classification. We found that the amplitude of the P300 was significantly greater in cases where the movement reached the target. Interestingly, we were able to classify the EEG signals evoked when observing the two classes of correct movements against each other with mean overall accuracy of 66.5 and 68.0% for the two data sets, with greater than chance levels of accuracy achieved for all participants. As a proof of concept, we have shown that it is possible to classify the EEG responses in observing these different correct movements against each other using single-trial EEG. This could be used as part of a learning-based BCI and opens a new door toward a more autonomous BCI navigation system.
Collapse
Affiliation(s)
- Christopher Wirth
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, United Kingdom
| | - Jake Toth
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, United Kingdom
| | - Mahnaz Arvaneh
- Automatic Control and Systems Engineering Department, University of Sheffield, Sheffield, United Kingdom
| |
Collapse
|
42
|
Krol LR, Haselager P, Zander TO. Cognitive and affective probing: a tutorial and review of active learning for neuroadaptive technology. J Neural Eng 2020; 17:012001. [DOI: 10.1088/1741-2552/ab5bb5] [Citation(s) in RCA: 18] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/04/2023]
|
43
|
Vidal F, Burle B, Hasbroucq T. Errors and Action Monitoring: Errare Humanum Est Sed Corrigere Possibile. Front Hum Neurosci 2020; 13:453. [PMID: 31998101 PMCID: PMC6962188 DOI: 10.3389/fnhum.2019.00453] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/29/2019] [Accepted: 12/09/2019] [Indexed: 01/12/2023] Open
Abstract
It was recognized long ago by Seneca through his famous "errare humanum est." that the human information processing system is intrinsically fallible. What is newer is the fact that, at least in sensorimotor information processing realized under time pressure, errors are largely dealt with by several (psycho)physiological-specific mechanisms: prevention, detection, inhibition, correction, and, if these mechanisms finally fail, strategic behavioral adjustments following errors. In this article, we review several datasets from laboratory experiments, showing that the human information processing system is well equipped not only to detect and correct errors when they occur but also to detect, inhibit, and correct them even before they fully develop. We argue that these (psycho)physiological mechanisms are important to consider when the brain works in everyday settings in order to render work systems more resilient to human errors and, thus, safer.
Collapse
Affiliation(s)
- Franck Vidal
- Aix-Marseille Université, CNRS, LNC UMR 7291, Marseille, France
| | | | | |
Collapse
|
44
|
Bevilacqua M, Perdikis S, Millan JDR. On Error-Related Potentials During Sensorimotor-Based Brain-Computer Interface: Explorations With a Pseudo-Online Brain-Controlled Speller. IEEE OPEN JOURNAL OF ENGINEERING IN MEDICINE AND BIOLOGY 2020; 1:17-22. [PMID: 35402943 PMCID: PMC8979633 DOI: 10.1109/ojemb.2019.2962879] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/05/2019] [Revised: 12/20/2019] [Accepted: 12/20/2019] [Indexed: 11/16/2022] Open
Abstract
Objective: Brain-computer interface (BCI) spelling is a promising communication solution for people in paralysis. Currently, BCIs suffer from imperfect decoding accuracy which calls for methods to handle spelling mistakes. Detecting error-related potentials (ErrPs) has been early identified as a potential remedy. Nevertheless, few works have studied the elicitation of ErrPs during engagement with other BCI tasks, especially when BCI feedback is provided continuously. Methods: Here, we test the possibility of correcting errors during pseudo-online Motor Imagery (MI) BCI spelling through ErrPs, and investigate whether BCI feedback hinders their generation. Ten subjects performed a series of MI spelling tasks with and without observing BCI feedback. Results: The average pseudo-online ErrP detection accuracy was found to be significantly above the chance level in both conditions and did not significantly differ between the two (74% with, and 78% without feedback). Conclusions: Our results support the possibility to detect ErrPs during MI-BCI spelling and suggest the absence of any BCI feedback-related interference.
Collapse
Affiliation(s)
- Michele Bevilacqua
- 1 Defitech Chair of Clinical NeuroengineeringCenter for Neuroprosthetics and Brain Mind Institute, EPFL CH-1202 Geneva Switzerland
| | - Serafeim Perdikis
- 2 Brain-Computer Interfaces and Neural Engineering Laboratory, School of Computer Science and Electronic EngineeringUniversity of Essex Colchester CO4 3SQ U.K
| | - Jose Del R Millan
- 3 Department of Electrical and Computer Engineering & Department of NeurologyThe University of Texas at Austin TX 78712 USA
| |
Collapse
|
45
|
Abstract
Big data and machine learning are having an impact on most aspects of modern life, from entertainment, commerce, and healthcare. Netflix knows which films and series people prefer to watch, Amazon knows which items people like to buy when and where, and Google knows which symptoms and conditions people are searching for. All this data can be used for very detailed personal profiling, which may be of great value for behavioral understanding and targeting but also has potential for predicting healthcare trends. There is great optimism that the application of artificial intelligence (AI) can provide substantial improvements in all areas of healthcare from diagnostics to treatment. It is generally believed that AI tools will facilitate and enhance human work and not replace the work of physicians and other healthcare staff as such. AI is ready to support healthcare personnel with a variety of tasks from administrative workflow to clinical documentation and patient outreach as well as specialized support such as in image analysis, medical device automation, and patient monitoring. In this chapter, some of the major applications of AI in healthcare will be discussed covering both the applications that are directly associated with healthcare and those in the healthcare value chain such as drug development and ambient assisted living.
Collapse
|
46
|
Abstract
The electroencephalogram (EEG) was invented almost 100 years ago and is still a method of choice for many research questions, even applications-from functional brain imaging in neuroscientific investigations during movement to real-time applications like brain-computer interfacing. This chapter gives some background information on the establishment and properties of the EEG. This chapter starts with a closer look at the sources of EEG at a micro or neuronal level, followed by recording techniques, types of electrodes, and common EEG artifacts. Then an overview on EEG phenomena, namely, spontaneous EEG and event-related potentials build the middle part of this chapter. The last part discusses brain signals, which are used in current BCI research, including short descriptions and examples of applications.
Collapse
Affiliation(s)
- Gernot R Müller-Putz
- Institute for Neural Engineering, Laboratory of Brain-Computer Interfaces, Graz University of Technology, Graz, Austria.
| |
Collapse
|
47
|
Abstract
Brain-computer interfaces (BCIs) have long been seen as control interfaces that translate changes in brain activity, produced either by means of a volitional modulation or in response to an external stimulation. However, recent trends in the BCI and neurofeedback research highlight passive monitoring of a user's brain activity in order to estimate cognitive load, attention level, perceived errors and emotions. Extraction of such higher order information from brain signals is seen as a gateway for facilitation of interaction between humans and intelligent systems. Particularly in the field of robotics, passive BCIs provide a promising channel for prediction of user's cognitive and affective state for development of a user-adaptive interaction. In this paper, we first illustrate the state of the art in passive BCI technology and then provide examples of BCI employment in human-robot interaction (HRI). We finally discuss the prospects and challenges in integration of passive BCIs in socially demanding HRI settings. This work intends to inform HRI community of the opportunities offered by passive BCI systems for enhancement of human-robot interaction while recognizing potential pitfalls.
Collapse
Affiliation(s)
- Maryam Alimardani
- Department of Cognitive Science and Artificial Intelligence, School of Humanities and Digital Sciences, Tilburg University, Tilburg, Netherlands
| | - Kazuo Hiraki
- Department of General Systems Studies, Graduate School of Arts and Sciences, The University of Tokyo, Tokyo, Japan
| |
Collapse
|
48
|
Wirth C, Dockree PM, Harty S, Lacey E, Arvaneh M. Towards error categorisation in BCI: single-trial EEG classification between different errors. J Neural Eng 2019; 17:016008. [DOI: 10.1088/1741-2552/ab53fe] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
|
49
|
Kuhner D, Fiederer L, Aldinger J, Burget F, Völker M, Schirrmeister R, Do C, Boedecker J, Nebel B, Ball T, Burgard W. A service assistant combining autonomous robotics, flexible goal formulation, and deep-learning-based brain–computer interfacing. ROBOTICS AND AUTONOMOUS SYSTEMS 2019; 116:98-113. [DOI: 10.1016/j.robot.2019.02.015] [Citation(s) in RCA: 14] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 08/30/2023]
|
50
|
Wilson NR, Sarma D, Wander JD, Weaver KE, Ojemann JG, Rao RPN. Cortical Topography of Error-Related High-Frequency Potentials During Erroneous Control in a Continuous Control Brain-Computer Interface. Front Neurosci 2019; 13:502. [PMID: 31191218 PMCID: PMC6541115 DOI: 10.3389/fnins.2019.00502] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2018] [Accepted: 04/30/2019] [Indexed: 11/13/2022] Open
Abstract
Brain–computer interfaces (BCIs) benefit greatly from performance feedback, but current systems lack automatic, task-independent feedback. Cortical responses elicited from user error have the potential to serve as state-based feedback to BCI decoders. To gain a better understanding of local error potentials, we investigate responsive cortical power underlying error-related potentials (ErrPs) from the human cortex during a one-dimensional center-out BCI task, tracking the topography of high-gamma (70–100 Hz) band power (HBP) specific to BCI error. We measured electrocorticography (ECoG) in three human subjects during dynamic, continuous control over BCI cursor velocity. Subjects used motor imagery and rest to move the cursor toward and subsequently dwell within a target region. We then identified and labeled epochs where the BCI decoder incorrectly moved the cursor in the direction opposite of the subject’s expectations (i.e., BCI error). We found increased HBP in various cortical areas 100–500 ms following BCI error with respect to epochs of correct, intended control. Significant responses were noted in primary somatosensory, motor, premotor, and parietal areas and generally regardless of whether the subject was using motor imagery or rest to move the cursor toward the target. Parts of somatosensory, temporal, and parietal areas exclusively had increased HBP when subjects were using motor imagery. In contrast, only part of the parietal cortex near the angular gyrus exclusively had an increase in HBP during rest. This investigation is, to our knowledge, the first to explore cortical fields changes in the context of continuous control in ECoG BCI. We present topographical changes in HBP characteristic specific to the generation of error. By focusing on continuous control, instead of on discrete control for simple selection, we investigate a more naturalistic setting and provide high ecological validity for characterizing error potentials. Such potentials could be considered as design elements for co-adaptive BCIs in the future as task-independent feedback to the decoder, allowing for more robust and individualized BCIs.
Collapse
Affiliation(s)
- Nile R Wilson
- Department of Bioengineering, University of Washington, Seattle, WA, United States
| | - Devapratim Sarma
- Department of Bioengineering, University of Washington, Seattle, WA, United States
| | - Jeremiah D Wander
- Department of Bioengineering, University of Washington, Seattle, WA, United States
| | - Kurt E Weaver
- Department of Radiology, University of Washington, Seattle, WA, United States
| | - Jeffrey G Ojemann
- Department of Neurological Surgery, University of Washington, Seattle, WA, United States
| | - Rajesh P N Rao
- Paul G. Allen School of Computer Science & Engineering, University of Washington, Seattle, WA, United States
| |
Collapse
|