1
|
Dillen A, Omidi M, Díaz MA, Ghaffari F, Roelands B, Vanderborght B, Romain O, De Pauw K. Evaluating the real-world usability of BCI control systems with augmented reality: a user study protocol. Front Hum Neurosci 2024; 18:1448584. [PMID: 39161850 PMCID: PMC11330773 DOI: 10.3389/fnhum.2024.1448584] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/13/2024] [Accepted: 07/18/2024] [Indexed: 08/21/2024] Open
Abstract
Brain-computer interfaces (BCI) enable users to control devices through their brain activity. Motor imagery (MI), the neural activity resulting from an individual imagining performing a movement, is a common control paradigm. This study introduces a user-centric evaluation protocol for assessing the performance and user experience of an MI-based BCI control system utilizing augmented reality. Augmented reality is employed to enhance user interaction by displaying environment-aware actions, and guiding users on the necessary imagined movements for specific device commands. One of the major gaps in existing research is the lack of comprehensive evaluation methodologies, particularly in real-world conditions. To address this gap, our protocol combines quantitative and qualitative assessments across three phases. In the initial phase, the BCI prototype's technical robustness is validated. Subsequently, the second phase involves a performance assessment of the control system. The third phase introduces a comparative analysis between the prototype and an alternative approach, incorporating detailed user experience evaluations through questionnaires and comparisons with non-BCI control methods. Participants engage in various tasks, such as object sorting, picking and placing, and playing a board game using the BCI control system. The evaluation procedure is designed for versatility, intending applicability beyond the specific use case presented. Its adaptability enables easy customization to meet the specific user requirements of the investigated BCI control application. This user-centric evaluation protocol offers a comprehensive framework for iterative improvements to the BCI prototype, ensuring technical validation, performance assessment, and user experience evaluation in a systematic and user-focused manner.
Collapse
Affiliation(s)
- Arnau Dillen
- Human Physiology and Sports Physiotherapy Research Group, Vrije Universiteit Brussel, Brussels, Belgium
- Équipes Traitement de l'Information et Systèmes, UMR 8051, CY Cergy Paris Université, École Nationale Supérieure de l'Électronique et de ses Applications (ENSEA), Centre national de la recherche scientifique (CNRS), Cergy, France
- Brussels Human Robotic Research Center (BruBotics), Vrije Universiteit Brussel, Brussels, Belgium
| | - Mohsen Omidi
- Brussels Human Robotic Research Center (BruBotics), Vrije Universiteit Brussel, Brussels, Belgium
- imec, Brussels, Belgium
| | - María Alejandra Díaz
- Human Physiology and Sports Physiotherapy Research Group, Vrije Universiteit Brussel, Brussels, Belgium
- Brussels Human Robotic Research Center (BruBotics), Vrije Universiteit Brussel, Brussels, Belgium
| | - Fakhreddine Ghaffari
- Équipes Traitement de l'Information et Systèmes, UMR 8051, CY Cergy Paris Université, École Nationale Supérieure de l'Électronique et de ses Applications (ENSEA), Centre national de la recherche scientifique (CNRS), Cergy, France
| | - Bart Roelands
- Human Physiology and Sports Physiotherapy Research Group, Vrije Universiteit Brussel, Brussels, Belgium
- Brussels Human Robotic Research Center (BruBotics), Vrije Universiteit Brussel, Brussels, Belgium
| | - Bram Vanderborght
- Brussels Human Robotic Research Center (BruBotics), Vrije Universiteit Brussel, Brussels, Belgium
- imec, Brussels, Belgium
| | - Olivier Romain
- Équipes Traitement de l'Information et Systèmes, UMR 8051, CY Cergy Paris Université, École Nationale Supérieure de l'Électronique et de ses Applications (ENSEA), Centre national de la recherche scientifique (CNRS), Cergy, France
| | - Kevin De Pauw
- Human Physiology and Sports Physiotherapy Research Group, Vrije Universiteit Brussel, Brussels, Belgium
- Brussels Human Robotic Research Center (BruBotics), Vrije Universiteit Brussel, Brussels, Belgium
| |
Collapse
|
2
|
Jeong JH, Cho JH, Lee BH, Lee SW. Real-Time Deep Neurolinguistic Learning Enhances Noninvasive Neural Language Decoding for Brain-Machine Interaction. IEEE TRANSACTIONS ON CYBERNETICS 2023; 53:7469-7482. [PMID: 36251899 DOI: 10.1109/tcyb.2022.3211694] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/16/2023]
Abstract
Electroencephalogram (EEG)-based brain-machine interface (BMI) has been utilized to help patients regain motor function and has recently been validated for its use in healthy people because of its ability to directly decipher human intentions. In particular, neurolinguistic research using EEGs has been investigated as an intuitive and naturalistic communication tool between humans and machines. In this study, the human mind directly decoded the neural languages based on speech imagery using the proposed deep neurolinguistic learning. Through real-time experiments, we evaluated whether BMI-based cooperative tasks between multiple users could be accomplished using a variety of neural languages. We successfully demonstrated a BMI system that allows a variety of scenarios, such as essential activity, collaborative play, and emotional interaction. This outcome presents a novel BMI frontier that can interact at the level of human-like intelligence in real time and extends the boundaries of the communication paradigm.
Collapse
|
3
|
Miao M, Yang Z, Zeng H, Zhang W, Xu B, Hu W. Explainable cross-task adaptive transfer learning for motor imagery EEG classification. J Neural Eng 2023; 20:066021. [PMID: 37963394 DOI: 10.1088/1741-2552/ad0c61] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/17/2023] [Accepted: 11/14/2023] [Indexed: 11/16/2023]
Abstract
Objective. In the field of motor imagery (MI) electroencephalography (EEG)-based brain-computer interfaces, deep transfer learning (TL) has proven to be an effective tool for solving the problem of limited availability in subject-specific data for the training of robust deep learning (DL) models. Although considerable progress has been made in the cross-subject/session and cross-device scenarios, the more challenging problem of cross-task deep TL remains largely unexplored.Approach. We propose a novel explainable cross-task adaptive TL method for MI EEG decoding. Firstly, similarity analysis and data alignment are performed for EEG data of motor execution (ME) and MI tasks. Afterwards, the MI EEG decoding model is obtained via pre-training with extensive ME EEG data and fine-tuning with partial MI EEG data. Finally, expected gradient-based post-hoc explainability analysis is conducted for the visualization of important temporal-spatial features.Main results. Extensive experiments are conducted on one large ME EEG High-Gamma dataset and two large MI EEG datasets (openBMI and GIST). The best average classification accuracy of our method reaches 80.00% and 72.73% for OpenBMI and GIST respectively, which outperforms several state-of-the-art algorithms. In addition, the results of the explainability analysis further validate the correlation between ME and MI EEG data and the effectiveness of ME/MI cross-task adaptation.Significance. This paper confirms that the decoding of MI EEG can be well facilitated by pre-existing ME EEG data, which largely relaxes the constraint of training samples for MI EEG decoding and is important in a practical sense.
Collapse
Affiliation(s)
- Minmin Miao
- School of Information Engineering, Huzhou University, Huzhou, People's Republic of China
- Zhejiang Province Key Laboratory of Smart Management & Application of Modern Agricultural Resources, Huzhou University, Huzhou, People's Republic of China
| | - Zhong Yang
- School of Information Engineering, Huzhou University, Huzhou, People's Republic of China
| | - Hong Zeng
- School of Instrument Science and Engineering, Southeast University, Nanjing, People's Republic of China
| | - Wenbin Zhang
- College of Computer and Information, Hohai University, Nanjing, People's Republic of China
| | - Baoguo Xu
- School of Instrument Science and Engineering, Southeast University, Nanjing, People's Republic of China
| | - Wenjun Hu
- School of Information Engineering, Huzhou University, Huzhou, People's Republic of China
- Zhejiang Province Key Laboratory of Smart Management & Application of Modern Agricultural Resources, Huzhou University, Huzhou, People's Republic of China
| |
Collapse
|
4
|
Bi J, Chu M. TDLNet: Transfer Data Learning Network for Cross-Subject Classification Based on Multiclass Upper Limb Motor Imagery EEG. IEEE Trans Neural Syst Rehabil Eng 2023; 31:3958-3967. [PMID: 37815969 DOI: 10.1109/tnsre.2023.3323509] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/12/2023]
Abstract
The limited number of brain-computer interface based on motor imagery (MI-BCI) instruction sets for different movements of single limbs makes it difficult to meet practical application requirements. Therefore, designing a single-limb, multi-category motor imagery (MI) paradigm and effectively decoding it is one of the important research directions in the future development of MI-BCI. Furthermore, one of the major challenges in MI-BCI is the difficulty of classifying brain activity across different individuals. In this article, the transfer data learning network (TDLNet) is proposed to achieve the cross-subject intention recognition for multiclass upper limb motor imagery. In TDLNet, the Transfer Data Module (TDM) is used to process cross-subject electroencephalogram (EEG) signals in groups and then fuse cross-subject channel features through two one-dimensional convolutions. The Residual Attention Mechanism Module (RAMM) assigns weights to each EEG signal channel and dynamically focuses on the EEG signal channels most relevant to a specific task. Additionally, a feature visualization algorithm based on occlusion signal frequency is proposed to qualitatively analyze the proposed TDLNet. The experimental results show that TDLNet achieves the best classification results on two datasets compared to CNN-based reference methods and transfer learning method. In the 6-class scenario, TDLNet obtained an accuracy of 65%±0.05 on the UML6 dataset and 63%±0.06 on the GRAZ dataset. The visualization results demonstrate that the proposed framework can produce distinct classifier patterns for multiple categories of upper limb motor imagery through signals of different frequencies. The ULM6 dataset is available at https://dx.doi.org/10.21227/8qw6-f578.
Collapse
|
5
|
Xiao D, Zhu F, Jiang J, Niu X. Leveraging natural cognitive systems in conjunction with ResNet50-BiGRU model and attention mechanism for enhanced medical image analysis and sports injury prediction. Front Neurosci 2023; 17:1273931. [PMID: 37795185 PMCID: PMC10546033 DOI: 10.3389/fnins.2023.1273931] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2023] [Accepted: 08/28/2023] [Indexed: 10/06/2023] Open
Abstract
Introduction In this study, we explore the potential benefits of integrating natural cognitive systems (medical professionals' expertise) and artificial cognitive systems (deep learning models) in the realms of medical image analysis and sports injury prediction. We focus on analyzing medical images of athletes to gain valuable insights into their health status. Methods To synergize the strengths of both natural and artificial cognitive systems, we employ the ResNet50-BiGRU model and introduce an attention mechanism. Our goal is to enhance the performance of medical image feature extraction and motion injury prediction. This integrated approach aims to achieve precise identification of anomalies in medical images, particularly related to muscle or bone damage. Results We evaluate the effectiveness of our method on four medical image datasets, specifically pertaining to skeletal and muscle injuries. We use performance indicators such as Peak Signal-to-Noise Ratio and Structural Similarity Index, confirming the robustness of our approach in sports injury analysis. Discussion Our research contributes significantly by providing an effective deep learning-driven method that harnesses both natural and artificial cognitive systems. By combining human expertise with advanced machine learning techniques, we offer a comprehensive understanding of athletes' health status. This approach holds potential implications for enhancing sports injury prevention, improving diagnostic accuracy, and tailoring personalized treatment plans for athletes, ultimately promoting better overall health and performance outcomes. Despite advancements in medical image analysis and sports injury prediction, existing systems often struggle to identify subtle anomalies and provide precise injury risk assessments, underscoring the necessity of a more integrated and comprehensive approach.
Collapse
Affiliation(s)
- Duo Xiao
- Ministry of Culture, Sports and Labor, Jiangxi Gannan Health Vocational College, Ganzhou, Jiangxi, China
| | - Fei Zhu
- Ministry of Culture, Sports and Labor, Jiangxi Gannan Health Vocational College, Ganzhou, Jiangxi, China
| | - Jian Jiang
- Gannan University of Science and Technology, Ganzhou, Jiangxi, China
| | - Xiaoqiang Niu
- Ministry of Culture, Sports and Labor, Jiangxi Gannan Health Vocational College, Ganzhou, Jiangxi, China
| |
Collapse
|
6
|
Mai X, Sheng X, Shu X, Ding Y, Zhu X, Meng J. A Calibration-Free Hybrid Approach Combining SSVEP and EOG for Continuous Control. IEEE Trans Neural Syst Rehabil Eng 2023; 31:3480-3491. [PMID: 37610901 DOI: 10.1109/tnsre.2023.3307814] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/25/2023]
Abstract
While SSVEP-BCI has been widely developed to control external devices, most of them rely on the discrete control strategy. The continuous SSVEP-BCI enables users to continuously deliver commands and receive real-time feedback from the devices, but it suffers from the transition state problem, a period the erroneous recognition, when users shift their gazes between targets. To resolve this issue, we proposed a novel calibration-free Bayesian approach by hybridizing SSVEP and electrooculography (EOG). First, canonical correlation analysis (CCA) was applied to detect the evoked SSVEPs, and saccade during the gaze shift was detected by EOG data using an adaptive threshold method. Then, the new target after the gaze shift was recognized based on a Bayesian optimization approach, which combined the detection of SSVEP and saccade together and calculated the optimized probability distribution of the targets. Eighteen healthy subjects participated in the offline and online experiments. The offline experiments showed that the proposed hybrid BCI had significantly higher overall continuous accuracy and shorter gaze-shifting time compared to FBCCA, CCA, MEC, and PSDA. In online experiments, the proposed hybrid BCI significantly outperformed CCA-based SSVEP-BCI in terms of continuous accuracy (77.61 ± 1.36%vs. 68.86 ± 1.08% and gaze-shifting time (0.93 ± 0.06s vs. 1.94 ± 0.08s). Additionally, participants also perceived a significant improvement over the CCA-based SSVEP-BCI when the newly proposed decoding approach was used. These results validated the efficacy of the proposed hybrid Bayesian approach for the BCI continuous control without any calibration. This study provides an effective framework for combining SSVEP and EOG, and promotes the potential applications of plug-and-play BCIs in continuous control.
Collapse
|
7
|
Sun C, Mou C. Survey on the research direction of EEG-based signal processing. Front Neurosci 2023; 17:1203059. [PMID: 37521708 PMCID: PMC10372445 DOI: 10.3389/fnins.2023.1203059] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/10/2023] [Accepted: 06/16/2023] [Indexed: 08/01/2023] Open
Abstract
Electroencephalography (EEG) is increasingly important in Brain-Computer Interface (BCI) systems due to its portability and simplicity. In this paper, we provide a comprehensive review of research on EEG signal processing techniques since 2021, with a focus on preprocessing, feature extraction, and classification methods. We analyzed 61 research articles retrieved from academic search engines, including CNKI, PubMed, Nature, IEEE Xplore, and Science Direct. For preprocessing, we focus on innovatively proposed preprocessing methods, channel selection, and data augmentation. Data augmentation is classified into conventional methods (sliding windows, segmentation and recombination, and noise injection) and deep learning methods [Generative Adversarial Networks (GAN) and Variation AutoEncoder (VAE)]. We also pay attention to the application of deep learning, and multi-method fusion approaches, including both conventional algorithm fusion and fusion between conventional algorithms and deep learning. Our analysis identifies 35 (57.4%), 18 (29.5%), and 37 (60.7%) studies in the directions of preprocessing, feature extraction, and classification, respectively. We find that preprocessing methods have become widely used in EEG classification (96.7% of reviewed papers) and comparative experiments have been conducted in some studies to validate preprocessing. We also discussed the adoption of channel selection and data augmentation and concluded several mentionable matters about data augmentation. Furthermore, deep learning methods have shown great promise in EEG classification, with Convolutional Neural Networks (CNNs) being the main structure of deep neural networks (92.3% of deep learning papers). We summarize and analyze several innovative neural networks, including CNNs and multi-structure fusion. However, we also identified several problems and limitations of current deep learning techniques in EEG classification, including inappropriate input, low cross-subject accuracy, unbalanced between parameters and time costs, and a lack of interpretability. Finally, we highlight the emerging trend of multi-method fusion approaches (49.2% of reviewed papers) and analyze the data and some examples. We also provide insights into some challenges of multi-method fusion. Our review lays a foundation for future studies to improve EEG classification performance.
Collapse
|
8
|
Jain A, Kumar L. EEG Cortical Source Feature based Hand Kinematics Decoding using Residual CNN-LSTM Neural Network. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-4. [PMID: 38082886 DOI: 10.1109/embc40787.2023.10341052] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Motor kinematics decoding (MKD) using brain signal is essential to develop Brain-computer interface (BCI) system for rehabilitation or prosthesis devices. Surface electroencephalogram (EEG) signal has been widely utilized for MKD. However, kinematic decoding from cortical sources is sparsely explored. In this work, the feasibility of hand kinematics decoding using EEG cortical source signals has been explored for grasp and lift task. In particular, pre-movement EEG segment is utilized. A residual convolutional neural network (CNN) - long short-term memory (LSTM) based kinematics decoding model is proposed that utilizes motor neural information present in pre-movement brain activity. Various EEG windows at 50 ms prior to movement onset, are utilized for hand kinematics decoding. Correlation value (CV) between actual and predicted hand kinematics is utilized as performance metric for source and sensor domain. The performance of the proposed deep learning model is compared in sensor and source domain. The results demonstrate the viability of hand kinematics decoding using pre-movement EEG cortical source data.
Collapse
|
9
|
Sen O, Sheehan AM, Raman PR, Khara KS, Khalifa A, Chatterjee B. Machine-Learning Methods for Speech and Handwriting Detection Using Neural Signals: A Review. SENSORS (BASEL, SWITZERLAND) 2023; 23:5575. [PMID: 37420741 DOI: 10.3390/s23125575] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Revised: 06/09/2023] [Accepted: 06/12/2023] [Indexed: 07/09/2023]
Abstract
Brain-Computer Interfaces (BCIs) have become increasingly popular in recent years due to their potential applications in diverse fields, ranging from the medical sector (people with motor and/or communication disabilities), cognitive training, gaming, and Augmented Reality/Virtual Reality (AR/VR), among other areas. BCI which can decode and recognize neural signals involved in speech and handwriting has the potential to greatly assist individuals with severe motor impairments in their communication and interaction needs. Innovative and cutting-edge advancements in this field have the potential to develop a highly accessible and interactive communication platform for these people. The purpose of this review paper is to analyze the existing research on handwriting and speech recognition from neural signals. So that the new researchers who are interested in this field can gain thorough knowledge in this research area. The current research on neural signal-based recognition of handwriting and speech has been categorized into two main types: invasive and non-invasive studies. We have examined the latest papers on converting speech-activity-based neural signals and handwriting-activity-based neural signals into text data. The methods of extracting data from the brain have also been discussed in this review. Additionally, this review includes a brief summary of the datasets, preprocessing techniques, and methods used in these studies, which were published between 2014 and 2022. This review aims to provide a comprehensive summary of the methodologies used in the current literature on neural signal-based recognition of handwriting and speech. In essence, this article is intended to serve as a valuable resource for future researchers who wish to investigate neural signal-based machine-learning methods in their work.
Collapse
Affiliation(s)
- Ovishake Sen
- Department of ECE, University of Florida, Gainesville, FL 32611, USA
| | - Anna M Sheehan
- Department of ECE, University of Florida, Gainesville, FL 32611, USA
| | - Pranay R Raman
- Department of ECE, University of Florida, Gainesville, FL 32611, USA
| | - Kabir S Khara
- Department of ECE, University of Florida, Gainesville, FL 32611, USA
| | - Adam Khalifa
- Department of ECE, University of Florida, Gainesville, FL 32611, USA
| | | |
Collapse
|
10
|
Wang P, Cao X, Zhou Y, Gong P, Yousefnezhad M, Shao W, Zhang D. A comprehensive review on motion trajectory reconstruction for EEG-based brain-computer interface. Front Neurosci 2023; 17:1086472. [PMID: 37332859 PMCID: PMC10272365 DOI: 10.3389/fnins.2023.1086472] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/01/2022] [Accepted: 05/03/2023] [Indexed: 06/20/2023] Open
Abstract
The advance in neuroscience and computer technology over the past decades have made brain-computer interface (BCI) a most promising area of neurorehabilitation and neurophysiology research. Limb motion decoding has gradually become a hot topic in the field of BCI. Decoding neural activity related to limb movement trajectory is considered to be of great help to the development of assistive and rehabilitation strategies for motor-impaired users. Although a variety of decoding methods have been proposed for limb trajectory reconstruction, there does not yet exist a review that covers the performance evaluation of these decoding methods. To alleviate this vacancy, in this paper, we evaluate EEG-based limb trajectory decoding methods regarding their advantages and disadvantages from a variety of perspectives. Specifically, we first introduce the differences in motor execution and motor imagery in limb trajectory reconstruction with different spaces (2D and 3D). Then, we discuss the limb motion trajectory reconstruction methods including experiment paradigm, EEG pre-processing, feature extraction and selection, decoding methods, and result evaluation. Finally, we expound on the open problem and future outlooks.
Collapse
Affiliation(s)
| | | | | | | | | | - Wei Shao
- *Correspondence: Wei Shao, ; Daoqiang Zhang,
| | | |
Collapse
|
11
|
Zhang C, Lin C, Leng Y, Fu Z, Cheng Y, Fu C. An Effective Head-Based HRI for 6D Robotic Grasping Using Mixed Reality. IEEE Robot Autom Lett 2023. [DOI: 10.1109/lra.2023.3261701] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 04/05/2023]
Affiliation(s)
- Chengjie Zhang
- Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems, Shenzhen, China
| | - Chengyu Lin
- Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems, Shenzhen, China
| | - Yuquan Leng
- Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems, Shenzhen, China
| | - Zezheng Fu
- Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems, Shenzhen, China
| | - Yaoyu Cheng
- Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems, Shenzhen, China
| | - Chenglong Fu
- Shenzhen Key Laboratory of Biomimetic Robotics and Intelligent Systems, Shenzhen, China
| |
Collapse
|
12
|
Guo R, Lin Y, Luo X, Gao X, Zhang S. A robotic arm control system with simultaneous and sequential modes combining eye-tracking with steady-state visual evoked potential in virtual reality environment. Front Neurorobot 2023; 17:1146415. [PMID: 37051328 PMCID: PMC10083338 DOI: 10.3389/fnbot.2023.1146415] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/17/2023] [Accepted: 03/10/2023] [Indexed: 03/29/2023] Open
Abstract
At present, single-modal brain-computer interface (BCI) still has limitations in practical application, such as low flexibility, poor autonomy, and easy fatigue for subjects. This study developed an asynchronous robotic arm control system based on steady-state visual evoked potentials (SSVEP) and eye-tracking in virtual reality (VR) environment, including simultaneous and sequential modes. For simultaneous mode, target classification was realized by decision-level fusion of electroencephalography (EEG) and eye-gaze. The stimulus duration for each subject was non-fixed, which was determined by an adjustable window method. Subjects could autonomously control the start and stop of the system using triple blink and eye closure, respectively. For sequential mode, no calibration was conducted before operation. First, subjects’ gaze area was obtained through eye-gaze, and then only few stimulus blocks began to flicker. Next, target classification was determined using EEG. Additionally, subjects could reject false triggering commands using eye closure. In this study, the system effectiveness was verified through offline experiment and online robotic-arm grasping experiment. Twenty subjects participated in offline experiment. For simultaneous mode, average ACC and ITR at the stimulus duration of 0.9 s were 90.50% and 60.02 bits/min, respectively. For sequential mode, average ACC and ITR at the stimulus duration of 1.4 s were 90.47% and 45.38 bits/min, respectively. Fifteen subjects successfully completed the online tasks of grabbing balls in both modes, and most subjects preferred the sequential mode. The proposed hybrid brain-computer interface (h-BCI) system could increase autonomy, reduce visual fatigue, meet individual needs, and improve the efficiency of the system.
Collapse
Affiliation(s)
- Rongxiao Guo
- School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, China
| | - Yanfei Lin
- School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, China
- *Correspondence: Yanfei Lin,
| | - Xi Luo
- School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing, China
| | - Xiaorong Gao
- Department of Biomedical Engineering, School of Medicine, Tsinghua University, Beijing, China
| | - Shangen Zhang
- School of Computer and Communication Engineering, University of Science and Technology Beijing, Beijing, China
| |
Collapse
|
13
|
Elbagoury BM, Vladareanu L, Vlădăreanu V, Salem AB, Travediu AM, Roushdy MI. A Hybrid Stacked CNN and Residual Feedback GMDH-LSTM Deep Learning Model for Stroke Prediction Applied on Mobile AI Smart Hospital Platform. SENSORS (BASEL, SWITZERLAND) 2023; 23:s23073500. [PMID: 37050561 PMCID: PMC10098561 DOI: 10.3390/s23073500] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/28/2022] [Revised: 02/15/2023] [Accepted: 03/07/2023] [Indexed: 06/12/2023]
Abstract
Artificial intelligence (AI) techniques for intelligent mobile computing in healthcare has opened up new opportunities in healthcare systems. Combining AI techniques with the existing Internet of Medical Things (IoMT) will enhance the quality of care that patients receive at home remotely and the successful establishment of smart living environments. Building a real AI for mobile AI in an integrated smart hospital environment is a challenging problem due to the complexities of receiving IoT medical sensors data, data analysis, and deep learning algorithm complexity programming for mobile AI engine implementation AI-based cloud computing complexities, especially when we tackle real-time environments of AI technologies. In this paper, we propose a new mobile AI smart hospital platform architecture for stroke prediction and emergencies. In addition, this research is focused on developing and testing different modules of integrated AI software based on XAI architecture, this is for the mobile health app as an independent expert system or as connected with a simulated environment of an AI-cloud-based solution. The novelty is in the integrated architecture and results obtained in our previous works and this extended research on hybrid GMDH and LSTM deep learning models for the proposed artificial intelligence and IoMT engine for mobile health edge computing technology. Its main goal is to predict heart-stroke disease. Current research is still missing a mobile AI system for heart/brain stroke prediction during patient emergency cases. This research work implements AI algorithms for stroke prediction and diagnosis. The hybrid AI in connected health is based on a stacked CNN and group handling method (GMDH) predictive analytics model, enhanced with an LSTM deep learning module for biomedical signals prediction. The techniques developed depend on the dataset of electromyography (EMG) signals, which provides a significant source of information for the identification of normal and abnormal motions in a stroke scenario. The resulting artificial intelligence mHealth app is an innovation beyond the state of the art and the proposed techniques achieve high accuracy as stacked CNN reaches almost 98% for stroke diagnosis. The GMDH neural network proves to be a good technique for monitoring the EMG signal of the same patient case with an average accuracy of 98.60% to an average of 96.68% of the signal prediction. Moreover, extending the GMDH model and a hybrid LSTM with dense layers deep learning model has improved significantly the prediction results that reach an average of 99%.
Collapse
Affiliation(s)
- Bassant M. Elbagoury
- Faculty of Computer and Information Sciences, Ain Shams University, Cairo 11566, Egypt
| | - Luige Vladareanu
- Institute of Solid Mechanics of the Romanian Academy, 010141 Bucharest, Romania
| | - Victor Vlădăreanu
- Institute of Solid Mechanics of the Romanian Academy, 010141 Bucharest, Romania
| | - Abdel Badeeh Salem
- Faculty of Computer and Information Sciences, Ain Shams University, Cairo 11566, Egypt
| | - Ana-Maria Travediu
- Institute of Solid Mechanics of the Romanian Academy, 010141 Bucharest, Romania
| | - Mohamed Ismail Roushdy
- Faculty of Computers and Information Technology, Future University in Egypt, New Cairo 11835, Egypt
| |
Collapse
|
14
|
Qiao Y, Luo J, Cui T, Liu H, Tang H, Zeng Y, Liu C, Li Y, Jian J, Wu J, Tian H, Yang Y, Ren TL, Zhou J. Soft Electronics for Health Monitoring Assisted by Machine Learning. NANO-MICRO LETTERS 2023; 15:66. [PMID: 36918452 PMCID: PMC10014415 DOI: 10.1007/s40820-023-01029-1] [Citation(s) in RCA: 14] [Impact Index Per Article: 14.0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/22/2022] [Accepted: 01/05/2023] [Indexed: 06/18/2023]
Abstract
Due to the development of the novel materials, the past two decades have witnessed the rapid advances of soft electronics. The soft electronics have huge potential in the physical sign monitoring and health care. One of the important advantages of soft electronics is forming good interface with skin, which can increase the user scale and improve the signal quality. Therefore, it is easy to build the specific dataset, which is important to improve the performance of machine learning algorithm. At the same time, with the assistance of machine learning algorithm, the soft electronics have become more and more intelligent to realize real-time analysis and diagnosis. The soft electronics and machining learning algorithms complement each other very well. It is indubitable that the soft electronics will bring us to a healthier and more intelligent world in the near future. Therefore, in this review, we will give a careful introduction about the new soft material, physiological signal detected by soft devices, and the soft devices assisted by machine learning algorithm. Some soft materials will be discussed such as two-dimensional material, carbon nanotube, nanowire, nanomesh, and hydrogel. Then, soft sensors will be discussed according to the physiological signal types (pulse, respiration, human motion, intraocular pressure, phonation, etc.). After that, the soft electronics assisted by various algorithms will be reviewed, including some classical algorithms and powerful neural network algorithms. Especially, the soft device assisted by neural network will be introduced carefully. Finally, the outlook, challenge, and conclusion of soft system powered by machine learning algorithm will be discussed.
Collapse
Affiliation(s)
- Yancong Qiao
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, 518107, People's Republic of China.
- Key Laboratory of Sensing Technology and Biomedical Instruments of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, 510275, People's Republic of China.
| | - Jinan Luo
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, 518107, People's Republic of China
- Key Laboratory of Sensing Technology and Biomedical Instruments of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, 510275, People's Republic of China
| | - Tianrui Cui
- School of Integrated Circuits and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, People's Republic of China
| | - Haidong Liu
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, 518107, People's Republic of China
- Key Laboratory of Sensing Technology and Biomedical Instruments of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, 510275, People's Republic of China
| | - Hao Tang
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, 518107, People's Republic of China
- Key Laboratory of Sensing Technology and Biomedical Instruments of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, 510275, People's Republic of China
| | - Yingfen Zeng
- School of Integrated Circuits and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, People's Republic of China
| | - Chang Liu
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, 518107, People's Republic of China
- Key Laboratory of Sensing Technology and Biomedical Instruments of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, 510275, People's Republic of China
| | - Yuanfang Li
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, 518107, People's Republic of China
- Key Laboratory of Sensing Technology and Biomedical Instruments of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, 510275, People's Republic of China
| | - Jinming Jian
- School of Integrated Circuits and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, People's Republic of China
| | - Jingzhi Wu
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, 518107, People's Republic of China
- Key Laboratory of Sensing Technology and Biomedical Instruments of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, 510275, People's Republic of China
| | - He Tian
- School of Integrated Circuits and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, People's Republic of China
| | - Yi Yang
- School of Integrated Circuits and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, People's Republic of China
| | - Tian-Ling Ren
- School of Integrated Circuits and Beijing National Research Center for Information Science and Technology (BNRist), Tsinghua University, Beijing, 100084, People's Republic of China.
| | - Jianhua Zhou
- School of Biomedical Engineering, Shenzhen Campus of Sun Yat-sen University, No. 66, Gongchang Road, Guangming District, Shenzhen, 518107, People's Republic of China.
- Key Laboratory of Sensing Technology and Biomedical Instruments of Guangdong Province, School of Biomedical Engineering, Sun Yat-sen University, Guangzhou, 510275, People's Republic of China.
| |
Collapse
|
15
|
Masengo G, Zhang X, Dong R, Alhassan AB, Hamza K, Mudaheranwa E. Lower limb exoskeleton robot and its cooperative control: A review, trends, and challenges for future research. Front Neurorobot 2023; 16:913748. [PMID: 36714152 PMCID: PMC9875327 DOI: 10.3389/fnbot.2022.913748] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/06/2022] [Accepted: 12/19/2022] [Indexed: 01/12/2023] Open
Abstract
Effective control of an exoskeleton robot (ER) using a human-robot interface is crucial for assessing the robot's movements and the force they produce to generate efficient control signals. Interestingly, certain surveys were done to show off cutting-edge exoskeleton robots. The review papers that were previously published have not thoroughly examined the control strategy, which is a crucial component of automating exoskeleton systems. As a result, this review focuses on examining the most recent developments and problems associated with exoskeleton control systems, particularly during the last few years (2017-2022). In addition, the trends and challenges of cooperative control, particularly multi-information fusion, are discussed.
Collapse
Affiliation(s)
- Gilbert Masengo
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China,Department of Mechanical Engineering, Rwanda Polytechnic/Integrated Polytechnic Regional College (IPRC) Karongi, Kigali, Rwanda,*Correspondence: Gilbert Masengo ✉
| | - Xiaodong Zhang
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China
| | - Runlin Dong
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China
| | - Ahmad B. Alhassan
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China
| | - Khaled Hamza
- School of Mechanical Engineering, Xi'an Jiaotong University, Xi'an, China,Shaanxi Key Laboratory of Intelligent Robot, Xi'an Jiaotong University, Xi'an, China
| | - Emmanuel Mudaheranwa
- Department of Mechanical Engineering, Rwanda Polytechnic/Integrated Polytechnic Regional College (IPRC) Karongi, Kigali, Rwanda,Department of Engineering, Cardiff University, Cardiff, United Kingdom
| |
Collapse
|
16
|
Gao W, Xu D, Li H, Du J, Wang G, Li D. Identification of adaptor proteins by incorporating deep learning and PSSM profiles. Methods 2023; 209:10-17. [PMID: 36427763 DOI: 10.1016/j.ymeth.2022.11.001] [Citation(s) in RCA: 3] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/13/2022] [Revised: 10/25/2022] [Accepted: 11/02/2022] [Indexed: 11/23/2022] Open
Abstract
Adaptor proteins, also known as signal transduction adaptor proteins, are important proteins in signal transduction pathways, and play a role in connecting signal proteins for signal transduction between cells. Studies have shown that adaptor proteins are closely related to some diseases, such as tumors and diabetes. Therefore, it is very meaningful to construct a relevant model to accurately identify adaptor proteins. In recent years, many studies have used a position-specific scoring matrix (PSSM) and neural network methods to identify adaptor proteins. However, ordinary neural network models cannot correlate the contextual information in PSSM profiles well, so these studies usually process 20×N (N > 20) PSSM into 20×20 dimensions, which results in the loss of a large amount of protein information; This research proposes an efficient method that combines one-dimensional convolution (1-D CNN) and a bidirectional long short-term memory network (biLSTM) to identify adaptor proteins. The complete PSSM profiles are the input of the model, and the complete information of the protein is retained during the training process. We perform cross-validation during model training and test the performance of the model on an independent test set; in the data set with 1224 adaptor proteins and 11,078 non-adaptor proteins, five indicators including specificity, sensitivity, accuracy, area under the receiver operating characteristic curve (AUC) metric and Matthews correlation coefficient (MCC), were employed to evaluate model performance. On the independent test set, the specificity, sensitivity, accuracy and MCC were 0.817, 0.865, 0.823 and 0.465, respectively. Those results show that our method is better than the state-of-the art methods. This study is committed to improve the accuracy of adaptor protein identification, and laid a foundation for further research on diseases related to adaptor protein. This research provided a new idea for the application of deep learning related models in bioinformatics and computational biology.
Collapse
Affiliation(s)
- Wentao Gao
- College of Information and Computer Engineering, Northeast Forestry University, Harbin 150000, China
| | - Dali Xu
- College of Information and Computer Engineering, Northeast Forestry University, Harbin 150000, China
| | - Hongfei Li
- College of Information and Computer Engineering, Northeast Forestry University, Harbin 150000, China
| | - Junping Du
- Beijing Key Laboratory of Intelligent Telecommunication Software and Multimedia, School of Computer Science, Beijing University of Posts and Telecommunications, Beijing, 100876, China
| | - Guohua Wang
- College of Information and Computer Engineering, Northeast Forestry University, Harbin 150000, China.
| | - Dan Li
- College of Information and Computer Engineering, Northeast Forestry University, Harbin 150000, China.
| |
Collapse
|
17
|
Cho JH, Jeong JH, Lee SW. NeuroGrasp: Real-Time EEG Classification of High-Level Motor Imagery Tasks Using a Dual-Stage Deep Learning Framework. IEEE TRANSACTIONS ON CYBERNETICS 2022; 52:13279-13292. [PMID: 34748509 DOI: 10.1109/tcyb.2021.3122969] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Brain-computer interfaces (BCIs) have been widely employed to identify and estimate a user's intention to trigger a robotic device by decoding motor imagery (MI) from an electroencephalogram (EEG). However, developing a BCI system driven by MI related to natural hand-grasp tasks is challenging due to its high complexity. Although numerous BCI studies have successfully decoded large body parts, such as the movement intention of both hands, arms, or legs, research on MI decoding of high-level behaviors such as hand grasping is essential to further expand the versatility of MI-based BCIs. In this study, we propose NeuroGrasp, a dual-stage deep learning framework that decodes multiple hand grasping from EEG signals under the MI paradigm. The proposed method effectively uses an EEG and electromyography (EMG)-based learning, such that EEG-based inference at test phase becomes possible. The EMG guidance during model training allows BCIs to predict hand grasp types from EEG signals accurately. Consequently, NeuroGrasp improved classification performance offline, and demonstrated a stable classification performance online. Across 12 subjects, we obtained an average offline classification accuracy of 0.68 (±0.09) in four-grasp-type classifications and 0.86 (±0.04) in two-grasp category classifications. In addition, we obtained an average online classification accuracy of 0.65 (±0.09) and 0.79 (±0.09) across six high-performance subjects. Because the proposed method has demonstrated a stable classification performance when evaluated either online or offline, in the future, we expect that the proposed method could contribute to different BCI applications, including robotic hands or neuroprosthetics for handling everyday objects.
Collapse
|
18
|
Kalafatovich J, Lee M, Lee SW. Decoding declarative memory process for predicting memory retrieval based on source localization. PLoS One 2022; 17:e0274101. [PMID: 36074790 PMCID: PMC9455842 DOI: 10.1371/journal.pone.0274101] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2021] [Accepted: 08/22/2022] [Indexed: 11/18/2022] Open
Abstract
Many studies have focused on understanding memory processes due to their importance in daily life. Differences in timing and power spectra of brain signals during encoding task have been linked to later remembered items and were recently used to predict memory retrieval performance. However, accuracies remain low when using non-invasive methods for acquiring brain signals, mainly due to the low spatial resolution. This study investigates the prediction of successful retrieval using estimated source activity corresponding either to cortical or subcortical structures through source localization. Electroencephalogram (EEG) signals were recorded while participants performed a declarative memory task. Frequency-time analysis was performed using signals from encoding and retrieval tasks to confirm the importance of neural oscillations and their relationship with later remembered and forgotten items. Significant differences in the power spectra between later remembered and forgotten items were found before and during the presentation of the stimulus in the encoding task. Source activity estimation revealed differences in the beta band power over the medial parietal and medial prefrontal areas prior to the presentation of the stimulus, and over the cuneus and lingual areas during the presentation of the stimulus. Additionally, there were significant differences during the stimuli presentation during the retrieval task. Prediction of later remembered items was performed using surface potentials and estimated source activity. The results showed that source localization increases classification performance compared to the one using surface potentials. These findings support the importance of incorporating spatial features of neural activity to improve the prediction of memory retrieval.
Collapse
Affiliation(s)
- Jenifer Kalafatovich
- Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea
| | - Minji Lee
- Department of Brain and Cognitive Engineering, Korea University, Seoul, Republic of Korea
| | - Seong-Whan Lee
- Department of Artificial Intelligence, Korea University, Seoul, Republic of Korea
- Department of Brain and Cognitive Engineering, Korea University, Seoul, Republic of Korea
- * E-mail:
| |
Collapse
|
19
|
Li J, Liang T, Zeng Z, Xu P, Chen Y, Guo Z, Liang Z, Xie L. Motion intention prediction of upper limb in stroke survivors using sEMG signal and attention mechanism. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103981] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
20
|
Jang SJ, Yang YJ, Ryun S, Kim JS, Chung CK, Jeong J. Decoding trajectories of imagined hand movement using electrocorticograms for brain-machine interface. J Neural Eng 2022; 19. [PMID: 35985293 DOI: 10.1088/1741-2552/ac8b37] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2022] [Accepted: 08/19/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Reaching hand movement is an important motor skill actively examined in brain-computer interface (BCI). Among various components of movement analyzed is the hand's trajectory, which describes the hand's continuous positions in three-dimensional space. While a large body of studies have investigated the decoding of real movements and the reconstruction of real hand movement trajectories from neural signals, fewer studies have attempted to decode the trajectory of imagined hand movement. To develop BCI systems for patients with hand motor dysfunctions, the systems essentially require to achieve movement-free control of external devices, which is only possible through successful decoding of purely imagined hand movement. APPROACH To achieve this goal, this study used a machine learning technique (i.e., the variational Bayesian least square) to analyze the electrocorticogram (ECoG) of eighteen epilepsy patients obtained from when they performed movement execution (ME) and kinesthetic movement imagination (KMI) of the reach-and-grasp hand action. MAIN RESULTS The variational Bayesian decoding model was able to successfully predict the imagined trajectories of hand movement significantly above chance level. The Pearson's correlation coefficient between imagined and predicted trajectories was 0.3393 and 0.4936 for the KMI (KMI trials only) and MEKMI paradigm (alternating trials of ME and KMI) respectively. SIGNIFICANCE This study demonstrated a high accuracy of prediction for trajectories of imagined hand movement, and more importantly, higher decoding accuracy of imagined trajectories in the MEKMI paradigm than in the KMI paradigm solely.
Collapse
Affiliation(s)
- Sang Jin Jang
- Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, 411 E16-1(YBS Building) Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu, Daejeon, South Korea 34141, Daejeon, Daejeon, 34141, Korea (the Republic of)
| | - Yu Jin Yang
- Seoul National University College of Natural Sciences, 103, Daehak-ro, Jongno-gu, Seoul, Republic of Korea, Seoul, 03080, Korea (the Republic of)
| | - Seokyun Ryun
- Seoul National University College of Natural Sciences, 103, Daehak-ro, Jongno-gu, Seoul, Republic of Korea, Seoul, 03080, Korea (the Republic of)
| | - June Sic Kim
- Seoul National University College of Natural Sciences, 103, Daehak-ro, Jongno-gu, Seoul, Republic of Korea, Seoul, 03080, Korea (the Republic of)
| | - Chun Kee Chung
- Seoul National University College of Natural Sciences, 103, Daehak-ro, Jongno-gu, Seoul, Republic of Korea, Seoul, 03080, Korea (the Republic of)
| | - Jaeseung Jeong
- Bio and Brain Engineering, Korea Advanced Institute of Science and Technology, 514 E16-1(YBS Building) Korea Advanced Institute of Science and Technology, 291 Daehak-ro, Yuseong-gu, Daejeon, South Korea 34141, Daejeon, 34141, Korea (the Republic of)
| |
Collapse
|
21
|
A Comprehensive Review of Endogenous EEG-Based BCIs for Dynamic Device Control. SENSORS 2022; 22:s22155802. [PMID: 35957360 PMCID: PMC9370865 DOI: 10.3390/s22155802] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 07/07/2022] [Revised: 07/23/2022] [Accepted: 07/30/2022] [Indexed: 11/28/2022]
Abstract
Electroencephalogram (EEG)-based brain–computer interfaces (BCIs) provide a novel approach for controlling external devices. BCI technologies can be important enabling technologies for people with severe mobility impairment. Endogenous paradigms, which depend on user-generated commands and do not need external stimuli, can provide intuitive control of external devices. This paper discusses BCIs to control various physical devices such as exoskeletons, wheelchairs, mobile robots, and robotic arms. These technologies must be able to navigate complex environments or execute fine motor movements. Brain control of these devices presents an intricate research problem that merges signal processing and classification techniques with control theory. In particular, obtaining strong classification performance for endogenous BCIs is challenging, and EEG decoder output signals can be unstable. These issues present myriad research questions that are discussed in this review paper. This review covers papers published until the end of 2021 that presented BCI-controlled dynamic devices. It discusses the devices controlled, EEG paradigms, shared control, stabilization of the EEG signal, traditional machine learning and deep learning techniques, and user experience. The paper concludes with a discussion of open questions and avenues for future work.
Collapse
|
22
|
Jeong JH, Cho JH, Lee YE, Lee SH, Shin GH, Kweon YS, Millán JDR, Müller KR, Lee SW. 2020 International brain-computer interface competition: A review. Front Hum Neurosci 2022; 16:898300. [PMID: 35937679 PMCID: PMC9354666 DOI: 10.3389/fnhum.2022.898300] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/17/2022] [Accepted: 07/01/2022] [Indexed: 11/16/2022] Open
Abstract
The brain-computer interface (BCI) has been investigated as a form of communication tool between the brain and external devices. BCIs have been extended beyond communication and control over the years. The 2020 international BCI competition aimed to provide high-quality neuroscientific data for open access that could be used to evaluate the current degree of technical advances in BCI. Although there are a variety of remaining challenges for future BCI advances, we discuss some of more recent application directions: (i) few-shot EEG learning, (ii) micro-sleep detection (iii) imagined speech decoding, (iv) cross-session classification, and (v) EEG(+ear-EEG) detection in an ambulatory environment. Not only did scientists from the BCI field compete, but scholars with a broad variety of backgrounds and nationalities participated in the competition to address these challenges. Each dataset was prepared and separated into three data that were released to the competitors in the form of training and validation sets followed by a test set. Remarkable BCI advances were identified through the 2020 competition and indicated some trends of interest to BCI researchers.
Collapse
Affiliation(s)
- Ji-Hoon Jeong
- School of Computer Science, Chungbuk National University, Cheongju, South Korea
| | - Jeong-Hyun Cho
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea
| | - Young-Eun Lee
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea
| | - Seo-Hyun Lee
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea
| | - Gi-Hwan Shin
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea
| | - Young-Seok Kweon
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea
| | - José del R. Millán
- Department of Electrical and Computer Engineering, University of Texas at Austin, Austin, TX, United States
| | - Klaus-Robert Müller
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea
- Machine Learning Group, Department of Computer Science, Berlin Institute of Technology, Berlin, Germany
- Max Planck Institute for Informatics, Saarbrucken, Germany
- Department of Artificial Intelligence, Korea University, Seoul, South Korea
| | - Seong-Whan Lee
- Department of Brain and Cognitive Engineering, Korea University, Seoul, South Korea
- Department of Artificial Intelligence, Korea University, Seoul, South Korea
| |
Collapse
|
23
|
Lee KW, Lee DH, Kim SJ, Lee SW. Decoding Neural Correlation of Language-Specific Imagined Speech using EEG Signals. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:1977-1980. [PMID: 36086641 DOI: 10.1109/embc48229.2022.9871721] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Speech impairments due to cerebral lesions and degenerative disorders can be devastating. For humans with severe speech deficits, imagined speech in the brain-computer interface has been a promising hope for reconstructing the neural signals of speech production. However, studies in the EEG-based imagined speech domain still have some limitations due to high variability in spatial and temporal information and low signal-to-noise ratio. In this paper, we investigated the neural signals for two groups of native speakers with two tasks with different languages, English and Chinese. Our assumption was that English, a non-tonal and phonogram-based language, would have spectral differences in neural computation compared to Chinese, a tonal and ideogram-based language. The results showed the significant difference in the relative power spectral density between English and Chinese in specific frequency band groups. Also, the spatial evaluation of Chinese native speakers in the theta band was distinctive during the imagination task. Hence, this paper would suggest the key spectral and spatial information of word imagination with specialized language while decoding the neural signals of speech. Clinical Relevance- Imagined speech-related studies lead to the development of assistive communication technology especially for patients with speech disorders such as aphasia due to brain damage. This study suggests significant spectral features by analyzing cross-language differences of EEG-based imagined speech using two widely used languages.
Collapse
|
24
|
Musellim S, Han DK, Jeong JH, Lee SW. Prototype-based Domain Generalization Framework for Subject-Independent Brain-Computer Interfaces. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:711-714. [PMID: 36086535 DOI: 10.1109/embc48229.2022.9871434] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Brain-computer interface (BCI) is challenging to use in practice due to the inter/intra-subject variability of electroencephalography (EEG). The BCI system, in general, necessitates a calibration technique to obtain subject/session-specific data in order to tune the model each time the system is utilized. This issue is acknowledged as a key hindrance to BCI, and a new strategy based on domain generalization has recently evolved to address it. In light of this, we've concentrated on developing an EEG classification framework that can be applied directly to data from unknown domains (i.e. subjects), using only data acquired from separate subjects previously. For this purpose, in this paper, we proposed a framework that employs the open-set recognition technique as an auxiliary task to learn subject-specific style features from the source dataset while helping the shared feature extractor with mapping the features of the unseen target dataset as a new unseen domain. Our aim is to impose cross-instance style in-variance in the same domain and reduce the open space risk on the potential unseen subject in order to improve the generalization ability of the shared feature extractor. Our experiments showed that using the domain information as an auxiliary network increases the generalization performance. Clinical relevance-This study suggests a strategy to improve the performance of the subject-independent BCI systems. Our framework can help to reduce the need for further calibration and can be utilized for a range of mental state monitoring tasks (e.g. neurofeedback, identification of epileptic seizures, and sleep disorders).
Collapse
|
25
|
Jeong JH, Kim KT, Kim DJ, Lee SJ, Kim H. Subject-Transfer Decoding using the Convolutional Neural Network for Motor Imagery-based Brain-Computer Interface. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2022; 2022:48-51. [PMID: 36086005 DOI: 10.1109/embc48229.2022.9871463] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/15/2023]
Abstract
Various pattern-recognition or machine learning-based methods have recently been developed to improve the accuracy of the motor imagery (MI)-based brain-computer interface (BCI). However, more research is needed to reduce the training time to apply it to the real-world environment. In this study, we propose a subject-transfer decoding method based on a convolutional neural network (CNN) which is robust even with a small number of training trials. The proposed CNN was pre-trained with other subjects' MI data and then fine-tuned to the target subject's training MI data. We evaluated the proposed method using the BCI competition IV data2a, which had the 4-class MIs. Consequently, on the same test dataset, with changing the number of training trials, the proposed method showed better accuracy than the self-training method, which used only the target subject's data for training, as averaged 86.54±7.78% (288 trials), 85.76 ±8.00% (240 trials), 84.65±8.11% (192 trials), and 83.29 ±8.25% (144 trials), respectively, which was 4.94% (288 trials), 6.10% (240 trials), 9.03% (192 trials), and 12.31% (144 trials)-point higher than the self-training method. Consequently, the proposed method was shown to be effective in maintaining classification accuracy even with the reduced training trials.
Collapse
|
26
|
Prabhakar SK, Rajaguru H, So K, Won DO. A Framework for Text Classification Using Evolutionary Contiguous Convolutional Neural Network and Swarm Based Deep Neural Network. Front Comput Neurosci 2022; 16:900885. [PMID: 35847966 PMCID: PMC9276958 DOI: 10.3389/fncom.2022.900885] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2022] [Accepted: 05/17/2022] [Indexed: 11/13/2022] Open
Abstract
To classify the texts accurately, many machine learning techniques have been utilized in the field of Natural Language Processing (NLP). For many pattern classification applications, great success has been obtained when implemented with deep learning models rather than using ordinary machine learning techniques. Understanding the complex models and their respective relationships within the data determines the success of such deep learning techniques. But analyzing the suitable deep learning methods, techniques, and architectures for text classification is a huge challenge for researchers. In this work, a Contiguous Convolutional Neural Network (CCNN) based on Differential Evolution (DE) is initially proposed and named as Evolutionary Contiguous Convolutional Neural Network (ECCNN) where the data instances of the input point are considered along with the contiguous data points in the dataset so that a deeper understanding is provided for the classification of the respective input, thereby boosting the performance of the deep learning model. Secondly, a swarm-based Deep Neural Network (DNN) utilizing Particle Swarm Optimization (PSO) with DNN is proposed for the classification of text, and it is named Swarm DNN. This model is validated on two datasets and the best results are obtained when implemented with the Swarm DNN model as it produced a high classification accuracy of 97.32% when tested on the BBC newsgroup text dataset and 87.99% when tested on 20 newsgroup text datasets. Similarly, when implemented with the ECCNN model, it produced a high classification accuracy of 97.11% when tested on the BBC newsgroup text dataset and 88.76% when tested on 20 newsgroup text datasets.
Collapse
Affiliation(s)
- Sunil Kumar Prabhakar
- Department of Artificial Intelligence Convergence, Hallym University, Chuncheon, South Korea
| | - Harikumar Rajaguru
- Department of ECE, Bannari Amman Institute of Technology, Sathyamangalam, India
| | - Kwangsub So
- Department of Artificial Intelligence Convergence, Hallym University, Chuncheon, South Korea
| | - Dong-Ok Won
- Department of Artificial Intelligence Convergence, Hallym University, Chuncheon, South Korea
- *Correspondence: Dong-Ok Won
| |
Collapse
|
27
|
Prabhakar SK, Rajaguru H, Kim C, Won DO. A Fusion-Based Technique With Hybrid Swarm Algorithm and Deep Learning for Biosignal Classification. Front Hum Neurosci 2022; 16:895761. [PMID: 35721347 PMCID: PMC9203681 DOI: 10.3389/fnhum.2022.895761] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/14/2022] [Accepted: 05/02/2022] [Indexed: 12/02/2022] Open
Abstract
The vital data about the electrical activities of the brain are carried by the electroencephalography (EEG) signals. The recordings of the electrical activity of brain neurons in a rhythmic and spontaneous manner from the scalp surface are measured by EEG. One of the most important aspects in the field of neuroscience and neural engineering is EEG signal analysis, as it aids significantly in dealing with the commercial applications as well. To uncover the highly useful information for neural classification activities, EEG studies incorporated with machine learning provide good results. In this study, a Fusion Hybrid Model (FHM) with Singular Value Decomposition (SVD) Based Estimation of Robust Parameters is proposed for efficient feature extraction of the biosignals and to understand the essential information it has for analyzing the brain functionality. The essential features in terms of parameter components are extracted using the developed hybrid model, and a specialized hybrid swarm technique called Hybrid Differential Particle Artificial Bee (HDPAB) algorithm is proposed for feature selection. To make the EEG more practical and to be used in a plethora of applications, the robust classification of these signals is necessary thereby relying less on the trained professionals. Therefore, the classification is done initially using the proposed Zero Inflated Poisson Mixture Regression Model (ZIPMRM) and then it is also classified with a deep learning methodology, and the results are compared with other standard machine learning techniques. This proposed flow of methodology is validated on a few standard Biosignal datasets, and finally, a good classification accuracy of 98.79% is obtained for epileptic dataset and 98.35% is obtained for schizophrenia dataset.
Collapse
Affiliation(s)
- Sunil Kumar Prabhakar
- Department of Artificial Intelligence Convergence, Hallym University, Chuncheon, South Korea
| | - Harikumar Rajaguru
- Department of Electronics and Communication Engineering, Bannari Amman Institute of Technology, Sathyamangalam, India
| | - Chulho Kim
- Department of Neurology, Chuncheon Sacred Heart Hospital, Chuncheon, South Korea
| | - Dong-Ok Won
- Department of Artificial Intelligence Convergence, Hallym University, Chuncheon, South Korea
- *Correspondence: Dong-Ok Won,
| |
Collapse
|
28
|
Ma W, Gong Y, Xue H, Liu Y, Lin X, Zhou G, Li Y. A lightweight and accurate double-branch neural network for four-class motor imagery classification. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103582] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|
29
|
Research on Robot Fuzzy Neural Network Motion System Based on Artificial Intelligence. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:4347772. [PMID: 35186062 PMCID: PMC8849933 DOI: 10.1155/2022/4347772] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/24/2021] [Revised: 12/29/2021] [Accepted: 01/10/2022] [Indexed: 11/18/2022]
Abstract
An intelligent controller based on a self-learning interval type-II fuzzy neural network is proposed to make the motion controller of the industrial intelligent robot with good adaptability. This controller has a parallel structure and contains an interval type-II fuzzy neural network and a conventional PD controller. For the design of the interval type-II fuzzy neural network, the interval type-II fuzzy set is established using the slave design method. In the design process of the interval type-II fuzzy set of the front piece, a dual sequence symmetric trapezoidal subordinate function arrangement method is proposed, which makes the self-learning law and stability analysis of the system in an analytic form and facilitates the implementation of the algorithm in hardware. In the design of the neural network self-learning law, a parametric self-learning algorithm based on sliding mode control theory is established to adjust the structural parameters of the interval type-II fuzzy neural network online, and the stability of the system is proved by using Lyapunov’s stability theorem. Three sets of validation simulation experiments are given in conjunction with the trajectory tracking problem of the Delta parallel robot. The simulation results show that, in the presence of system uncertainty, the intelligent controller based on interval self-learning interval type-II fuzzy neural network can significantly improve the trajectory tracking accuracy and robustness of the system and make the control system highly adaptable to the environment. Experiments of intelligent control system based on self-learning interval type-II fuzzy neural network and experiments of reusable particle swarm optimal motion planning method are designed, and the effectiveness of the intelligent control system and motion planning method is verified on the experimental platform. The experimental results show that the intelligent control system based on the self-learning interval type-II fuzzy neural network can effectively improve the accuracy and stability of robot trajectory tracking control, and the reusable particle swarm optimal motion planning method can quickly solve the robot motion planning problem with complex constraints online.
Collapse
|
30
|
Kwak Y, Song WJ, Kim SE. FGANet: fNIRS-guided Attention Network for Hybrid EEG-fNIRS Brain-Computer Interfaces. IEEE Trans Neural Syst Rehabil Eng 2022; 30:329-339. [PMID: 35130163 DOI: 10.1109/tnsre.2022.3149899] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Non-invasive brain-computer interfaces (BCIs) have been widely used for neural decoding, linking neural signals to control devices. Hybrid BCI systems using electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have received significant attention for overcoming the limitations of EEG- and fNIRS-standalone BCI systems. However, most hybrid EEG-fNIRS BCI studies have focused on late fusion because of discrepancies in their temporal resolutions and recording locations. Despite the enhanced performance of hybrid BCIs, late fusion methods have difficulty in extracting correlated features in both EEG and fNIRS signals. Therefore, in this study, we proposed a deep learning-based early fusion structure, which combines two signals before the fully-connected layer, called the fNIRS-guided attention network (FGANet). First, 1D EEG and fNIRS signals were converted into 3D EEG and fNIRS tensors to spatially align EEG and fNIRS signals at the same time point. The proposed fNIRS-guided attention layer extracted a joint representation of EEG and fNIRS tensors based on neurovascular coupling, in which the spatially important regions were identified from fNIRS signals, and detailed neural patterns were extracted from EEG signals. Finally, the final prediction was obtained by weighting the sum of the prediction scores of the EEG and fNIRS-guided attention features to alleviate performance degradation owing to delayed fNIRS response. In the experimental results, the FGANet significantly outperformed the EEG-standalone network. Furthermore, the FGANet has 4.0% and 2.7% higher accuracy than the state-of-the-art algorithms in mental arithmetic and motor imagery tasks, respectively.
Collapse
|
31
|
A multiscale siamese convolutional neural network with cross-channel fusion for motor imagery decoding. J Neurosci Methods 2022; 367:109426. [PMID: 34902364 DOI: 10.1016/j.jneumeth.2021.109426] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/15/2021] [Revised: 10/28/2021] [Accepted: 11/22/2021] [Indexed: 11/22/2022]
Abstract
BACKGROUND Recently, convolutional neural networks (CNN) are widely applied in motor imagery electroencephalography (MI-EEG) signal classification tasks. However, a simple CNN framework is challenging to satisfy the complex MI-EEG signal decoding. NEW METHOD In this study, we propose a multiscale Siamese convolutional neural network with cross-channel fusion (MSCCF-Net) for MI-EEG classification tasks. The proposed network consists of three parts: Siamese cross-channel fusion streams, similarity module and classification module. Each Siamese cross-channel fusion stream contains multiple branches, and each branch is supplemented by cross-channel fusion modules to improve multiscale temporal feature representation capability. The similarity module is adopted to measure the feature similarity between multiple branches. At the same time, the classification module provides a strong constraint to classify the features from all Siamese cross-channel fusion streams. The combination of the similarity module and classification module constitutes a new joint training strategy to further optimize the network performance. RESULTS The experiment is conducted on the public BCI Competition IV 2a and 2b datasets, and the results show that the proposed network achieves an average accuracy of 87.36% and 87.33%, respectively. COMPARISON WITH EXISTING METHODS AND CONCLUSIONS The proposed network adopts cross-channel fusion to learn multiscale temporal characteristics and joint training strategy to optimize the training process. Therefore, the performance outperforms other state-of-the-art MI-EEG signal classification methods.
Collapse
|
32
|
Dillen A, Steckelmacher D, Efthymiadis K, Langlois K, De Beir A, Marušič U, Vanderborght B, Nowé A, Meeusen R, Ghaffari F, Romain O, De Pauw K. Deep learning for biosignal control: insights from basic to real-time methods with recommendations. J Neural Eng 2022; 19. [PMID: 35086076 DOI: 10.1088/1741-2552/ac4f9a] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2021] [Accepted: 01/27/2022] [Indexed: 11/11/2022]
Abstract
Biosignal control is an interaction modality that allows users to interact with electronic devices by decoding the biological signals emanating from the movements or thoughts of the user. This manner of interaction with devices can enhance the sense of agency for users and enable persons suffering from a paralyzing condition to interact with everyday devices that would otherwise be challenging for them to use. It can also improve control of prosthetic devices and exoskeletons by making the interaction feel more natural and intuitive. However, with the current state of the art, several issues still need to be addressed to reliably decode user intent from biosignals and provide an improved user experience over other interaction modalities. One solution is to leverage advances in Deep Learning (DL) methods to provide more reliable decoding at the expense of added computational complexity. This scoping review introduces the basic concepts of DL and assists readers in deploying DL methods to a real-time control system that should operate under real-world conditions. The scope of this review covers any electronic device, but with an emphasis on robotic devices, as this is the most active area of research in biosignal control. We review the literature pertaining to the implementation and evaluation of control systems that incorporate DL to identify the main gaps and issues in the field, and formulate suggestions on how to mitigate them. Additionally, we formulate guidelines on the best approach to designing, implementing and evaluating research prototypes that use DL in their biosignal control systems.
Collapse
Affiliation(s)
- Arnau Dillen
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | | | | | - Kevin Langlois
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | - Albert De Beir
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | - Uroš Marušič
- Alma Mater Europaea - Evropski Center Maribor, Slovenska ulica 17, Maribor, Maribor, 2000, SLOVENIA
| | - Bram Vanderborght
- Vrije Universiteit Brussel, Faculty of Applied Sciences, Brussel, Brussel, 1050, BELGIUM
| | - Ann Nowé
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | - Romain Meeusen
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| | - Fakhreddine Ghaffari
- Equipe Traitement de l'Information et Systèmes, CY Cergy Paris University, 6 Rue du Ponceau, Cergy-Pontoise, 95000 , FRANCE
| | - Olivier Romain
- Equipe Traitement de l'Information et Systèmes, CY Cergy Paris University, 6 Rue du Ponceau, Cergy-Pontoise, 95000 , FRANCE
| | - Kevin De Pauw
- Vrije Universiteit Brussel, Pleinlaan 2, Brussel, Brussel, 1050, BELGIUM
| |
Collapse
|
33
|
Lee DY, Jeong JH, Lee BH, Lee SW. Motor Imagery Classification Using Inter-Task Transfer Learning via A Channel-Wise Variational Autoencoder-based Convolutional Neural Network. IEEE Trans Neural Syst Rehabil Eng 2022; 30:226-237. [PMID: 35041605 DOI: 10.1109/tnsre.2022.3143836] [Citation(s) in RCA: 9] [Impact Index Per Article: 4.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Highly sophisticated control based on a brain-computer interface (BCI) requires decoding kinematic information from brain signals. The forearm is a region of the upper limb that is often used in everyday life, but intuitive movements within the same limb have rarely been investigated in previous BCI studies. In this study, we focused on various forearm movement decoding from electroencephalography (EEG) signals using a small number of samples. Ten healthy participants took part in an experiment and performed motor execution (ME) and motor imagery (MI) of the intuitive movement tasks (Dataset I). We propose a convolutional neural network using a channel-wise variational autoencoder (CVNet) based on inter-task transfer learning. We approached that training the reconstructed ME-EEG signals together will also achieve more sufficient classification performance with only a small amount of MI-EEG signals. The proposed CVNet was validated on our own Dataset I and a public dataset, BNCI Horizon 2020 (Dataset II). The classification accuracies of various movements are confirmed to be 0.83 (±0.04) and 0.69 (±0.04) for Dataset I and II, respectively. The results show that the proposed method exhibits performance increases of approximately 0.09~0.27 and 0.08~0.24 compared with the conventional models for Dataset I and II, respectively. The outcomes suggest that the training model for decoding imagined movements can be performed using data from ME and a small number of data samples from MI. Hence, it is presented the feasibility of BCI learning strategies that can sufficiently learn deep learning with a few amount of calibration dataset and time only, with stable performance.
Collapse
|
34
|
Chacon-Murguia MI, Rivas-Posada E. A CNN-based modular classification scheme for motor imagery using a novel EEG sampling protocol suitable for IoT healthcare systems. Neural Comput Appl 2022. [DOI: 10.1007/s00521-021-06716-x] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
35
|
Lee YE, Shin GH, Lee M, Lee SW. Mobile BCI dataset of scalp- and ear-EEGs with ERP and SSVEP paradigms while standing, walking, and running. Sci Data 2021; 8:315. [PMID: 34930915 PMCID: PMC8688416 DOI: 10.1038/s41597-021-01094-4] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/20/2021] [Accepted: 11/08/2021] [Indexed: 11/24/2022] Open
Abstract
We present a mobile dataset obtained from electroencephalography (EEG) of the scalp and around the ear as well as from locomotion sensors by 24 participants moving at four different speeds while performing two brain-computer interface (BCI) tasks. The data were collected from 32-channel scalp-EEG, 14-channel ear-EEG, 4-channel electrooculography, and 9-channel inertial measurement units placed at the forehead, left ankle, and right ankle. The recording conditions were as follows: standing, slow walking, fast walking, and slight running at speeds of 0, 0.8, 1.6, and 2.0 m/s, respectively. For each speed, two different BCI paradigms, event-related potential and steady-state visual evoked potential, were recorded. To evaluate the signal quality, scalp- and ear-EEG data were qualitatively and quantitatively validated during each speed. We believe that the dataset will facilitate BCIs in diverse mobile environments to analyze brain activities and evaluate the performance quantitatively for expanding the use of practical BCIs.
Collapse
Affiliation(s)
- Young-Eun Lee
- grid.222754.40000 0001 0840 2678Korea University, Department of Brain and Cognitive Engineering, Seoul, 02841 Republic of Korea
| | - Gi-Hwan Shin
- grid.222754.40000 0001 0840 2678Korea University, Department of Brain and Cognitive Engineering, Seoul, 02841 Republic of Korea
| | - Minji Lee
- grid.222754.40000 0001 0840 2678Korea University, Department of Brain and Cognitive Engineering, Seoul, 02841 Republic of Korea
| | - Seong-Whan Lee
- Korea University, Department of Brain and Cognitive Engineering, Seoul, 02841, Republic of Korea. .,Korea University, Department of Artificial Intelligence, Seoul, 02841, Republic of Korea.
| |
Collapse
|
36
|
Zheng L, Wang Z, Liang J, Luo S, Tian S. Effective compression and classification of ECG arrhythmia by singular value decomposition. BIOMEDICAL ENGINEERING ADVANCES 2021. [DOI: 10.1016/j.bea.2021.100013] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/20/2022] Open
|
37
|
Emerging trends in BCI-robotics for motor control and rehabilitation. CURRENT OPINION IN BIOMEDICAL ENGINEERING 2021. [DOI: 10.1016/j.cobme.2021.100354] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/19/2022]
|
38
|
Chen L, Chen P, Zhao S, Luo Z, Chen W, Pei Y, Zhao H, Jiang J, Xu M, Yan Y, Yin E. Adaptive asynchronous control system of robotic arm based on augmented reality-assisted brain-computer interface. J Neural Eng 2021; 18. [PMID: 34654000 DOI: 10.1088/1741-2552/ac3044] [Citation(s) in RCA: 16] [Impact Index Per Article: 5.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/12/2021] [Accepted: 10/15/2021] [Indexed: 11/12/2022]
Abstract
Objective. Brain-controlled robotic arms have shown broad application prospects with the development of robotics, science and information decoding. However, disadvantages, such as poor flexibility restrict its wide application.Approach. In order to alleviate these drawbacks, this study proposed a robotic arm asynchronous control system based on steady-state visual evoked potential (SSVEP) in an augmented reality (AR) environment. In the AR environment, the participants were able to concurrently see the robot arm and visual stimulation interface through the AR device. Therefore, there was no need to switch attention frequently between the visual stimulation interface and the robotic arm. This study proposed a multi-template algorithm based on canonical correlation analysis and task-related component analysis to identify 12 targets. An optimization strategy based on dynamic window was adopted to adjust the duration of visual stimulation adaptively.Main results. Experimental results of this study found that the high-frequency SSVEP-based brain-computer interface (BCI) realized the switch of the system state, which controlled the robotic arm asynchronously. The average accuracy of the offline experiment was 94.97%, whereas the average information translate rate was 67.37 ± 14.27 bits·min-1. The online results from ten healthy subjects showed that the average selection time of a single online command was 2.04 s, which effectively reduced the visual fatigue of the subjects. Each subject could quickly complete the puzzle task.Significance. The experimental results demonstrated the feasibility and potential of this human-computer interaction strategy and provided new ideas for BCI-controlled robots.
Collapse
Affiliation(s)
- Lingling Chen
- School of Artificial Intelligence and Data Science, Hebei University of Technology, Tianjin 300130, People's Republic of China.,Engineering Research Center of Intelligent Rehabilitation Device and Detection Technology Ministry of Education, Tianjin 300130, People's Republic of China
| | - Pengfei Chen
- School of Artificial Intelligence and Data Science, Hebei University of Technology, Tianjin 300130, People's Republic of China.,Engineering Research Center of Intelligent Rehabilitation Device and Detection Technology Ministry of Education, Tianjin 300130, People's Republic of China.,Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin 300450, People's Republic of China
| | - Shaokai Zhao
- Defense Innovation Institute, Academy of Military Sciences (AMS), Beijing 100071, People's Republic of China.,Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin 300450, People's Republic of China
| | - Zhiguo Luo
- Defense Innovation Institute, Academy of Military Sciences (AMS), Beijing 100071, People's Republic of China.,Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin 300450, People's Republic of China
| | - Wei Chen
- National Research Center for Rehabilitation Technical Aids, Beijing 100176, People's Republic of China
| | - Yu Pei
- Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin 300450, People's Republic of China
| | - Hongyu Zhao
- Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin 300450, People's Republic of China.,East China University of Science and Technology, Shanghai 200237, People's Republic of China
| | - Jing Jiang
- National Key Laboratory of Human Factors Engineering, China Astronaut Research and Training Center, Beijing 100094, People's Republic of China
| | - Minpeng Xu
- Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin 300450, People's Republic of China.,Tianjin University, Tianjin 300072, People's Republic of China
| | - Ye Yan
- Defense Innovation Institute, Academy of Military Sciences (AMS), Beijing 100071, People's Republic of China.,Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin 300450, People's Republic of China
| | - Erwei Yin
- Defense Innovation Institute, Academy of Military Sciences (AMS), Beijing 100071, People's Republic of China.,Tianjin Artificial Intelligence Innovation Center (TAIIC), Tianjin 300450, People's Republic of China
| |
Collapse
|
39
|
Ko LW, Stevenson C, Chang WC, Yu KH, Chi KC, Chen YJ, Chen CH. Integrated Gait Triggered Mixed Reality and Neurophysiological Monitoring as a Framework for Next-Generation Ambulatory Stroke Rehabilitation. IEEE Trans Neural Syst Rehabil Eng 2021; 29:2435-2444. [PMID: 34748494 DOI: 10.1109/tnsre.2021.3125946] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Brain stroke affects millions of people in the world every year, with 50 to 60 percent of stroke survivors suffering from functional disabilities, for which early and sustained post-stroke rehabilitation is highly recommended. However, approximately one third of stroke patients do not receive early in hospital rehabilitation programs due to insufficient medical facilities or lack of motivation. Gait triggered mixed reality (GTMR) is a cognitive-motor dual task with multisensory feedback tailored for lower-limb post-stroke rehabilitation, which we propose as a potential method for addressing these rehabilitation challenges. Simultaneous gait and EEG data from nine stroke patients was recorded and analyzed to assess the applicability of GTMR to different stroke patients, determine any impacts of GTMR on patients, and better understand brain dynamics as stroke patients perform different rehabilitation tasks. Walking cadence improved significantly for stroke patients and lower-limb movement induced alpha band power suppression during GTMR tasks. The brain dynamics and gait performance across different severities of stroke motor deficits was also assessed; the intensity of walking induced event related desynchronization (ERD) was found to be related to motor deficits, as classified by Brunnstrom stage. In particular, stronger lower-limb movement induced ERD during GTMR rehabilitation tasks was found for patients with moderate motor deficits (Brunnstrom stage IV). This investigation demonstrates the efficacy of the GTMR paradigm for enhancing lower-limb rehabilitation, explores the neural activities of cognitive-motor tasks in different stages of stroke, and highlights the potential for joining enhanced rehabilitation and real-time neural monitoring for superior stroke rehabilitation.
Collapse
|
40
|
Jeong JH, Choi JH, Kim KT, Lee SJ, Kim DJ, Kim HM. Multi-Domain Convolutional Neural Networks for Lower-Limb Motor Imagery Using Dry vs. Wet Electrodes. SENSORS 2021; 21:s21196672. [PMID: 34640992 PMCID: PMC8513081 DOI: 10.3390/s21196672] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/25/2021] [Revised: 10/04/2021] [Accepted: 10/05/2021] [Indexed: 11/29/2022]
Abstract
Motor imagery (MI) brain–computer interfaces (BCIs) have been used for a wide variety of applications due to their intuitive matching between the user’s intentions and the performance of tasks. Applying dry electroencephalography (EEG) electrodes to MI BCI applications can resolve many constraints and achieve practicality. In this study, we propose a multi-domain convolutional neural networks (MD-CNN) model that learns subject-specific and electrode-dependent EEG features using a multi-domain structure to improve the classification accuracy of dry electrode MI BCIs. The proposed MD-CNN model is composed of learning layers for three domain representations (time, spatial, and phase). We first evaluated the proposed MD-CNN model using a public dataset to confirm 78.96% classification accuracy for multi-class classification (chance level accuracy: 30%). After that, 10 healthy subjects participated and performed three classes of MI tasks related to lower-limb movement (gait, sitting down, and resting) over two sessions (dry and wet electrodes). Consequently, the proposed MD-CNN model achieved the highest classification accuracy (dry: 58.44%; wet: 58.66%; chance level accuracy: 43.33%) with a three-class classifier and the lowest difference in accuracy between the two electrode types (0.22%, d = 0.0292) compared with the conventional classifiers (FBCSP, EEGNet, ShallowConvNet, and DeepConvNet) that used only a single domain. We expect that the proposed MD-CNN model could be applied for developing robust MI BCI systems with dry electrodes.
Collapse
Affiliation(s)
- Ji-Hyeok Jeong
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
- Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, Korea
| | - Jun-Hyuk Choi
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
- Division of Bio-Medical Science & Technology, KIST School, Korea University of Science and Technology, Seoul 02792, Korea
| | - Keun-Tae Kim
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
| | - Song-Joo Lee
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
- Division of Bio-Medical Science & Technology, KIST School, Korea University of Science and Technology, Seoul 02792, Korea
| | - Dong-Joo Kim
- Department of Brain and Cognitive Engineering, Korea University, Seoul 02841, Korea
- Department of Neurology, Korea University College of Medicine, Seoul 02841, Korea
- Department of Artificial Intelligence, Korea University, Seoul 02841, Korea
- Correspondence: (D.-J.K.); (H.-M.K.)
| | - Hyung-Min Kim
- Biomedical Research Division, Bionics Research Center, Korea Institute of Science and Technology, Seoul 02792, Korea; (J.-H.J.); (J.-H.C.); (K.-T.K.); (S.-J.L.)
- Division of Bio-Medical Science & Technology, KIST School, Korea University of Science and Technology, Seoul 02792, Korea
- Correspondence: (D.-J.K.); (H.-M.K.)
| |
Collapse
|
41
|
Ma C, Guo W, Zhang H, Samuel OW, Ji X, Xu L, Li G. A Novel and Efficient Feature Extraction Method for Deep Learning Based Continuous Estimation. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3097257] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
42
|
Deep Learning-Based Stroke Disease Prediction System Using Real-Time Bio Signals. SENSORS 2021; 21:s21134269. [PMID: 34206540 PMCID: PMC8271462 DOI: 10.3390/s21134269] [Citation(s) in RCA: 11] [Impact Index Per Article: 3.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 05/10/2021] [Revised: 06/15/2021] [Accepted: 06/16/2021] [Indexed: 11/17/2022]
Abstract
The emergence of an aging society is inevitable due to the continued increases in life expectancy and decreases in birth rate. These social changes require new smart healthcare services for use in daily life, and COVID-19 has also led to a contactless trend necessitating more non-face-to-face health services. Due to the improvements that have been achieved in healthcare technologies, an increasing number of studies have attempted to predict and analyze certain diseases in advance. Research on stroke diseases is actively underway, particularly with the aging population. Stroke, which is fatal to the elderly, is a disease that requires continuous medical observation and monitoring, as its recurrence rate and mortality rate are very high. Most studies examining stroke disease to date have used MRI or CT images for simple classification. This clinical approach (imaging) is expensive and time-consuming while requiring bulky equipment. Recently, there has been increasing interest in using non-invasive measurable EEGs to compensate for these shortcomings. However, the prediction algorithms and processing procedures are both time-consuming because the raw data needs to be separated before the specific attributes can be obtained. Therefore, in this paper, we propose a new methodology that allows for the immediate application of deep learning models on raw EEG data without using the frequency properties of EEG. This proposed deep learning-based stroke disease prediction model was developed and trained with data collected from real-time EEG sensors. We implemented and compared different deep-learning models (LSTM, Bidirectional LSTM, CNN-LSTM, and CNN-Bidirectional LSTM) that are specialized in time series data classification and prediction. The experimental results confirmed that the raw EEG data, when wielded by the CNN-bidirectional LSTM model, can predict stroke with 94.0% accuracy with low FPR (6.0%) and FNR (5.7%), thus showing high confidence in our system. These experimental results demonstrate the feasibility of non-invasive methods that can easily measure brain waves alone to predict and monitor stroke diseases in real time during daily life. These findings are expected to lead to significant improvements for early stroke detection with reduced cost and discomfort compared to other measuring techniques.
Collapse
|
43
|
Stieger JR, Engel SA, Suma D, He B. Benefits of deep learning classification of continuous noninvasive brain-computer interface control. J Neural Eng 2021; 18. [PMID: 34038873 DOI: 10.1088/1741-2552/ac0584] [Citation(s) in RCA: 19] [Impact Index Per Article: 6.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/16/2021] [Accepted: 05/26/2021] [Indexed: 11/12/2022]
Abstract
Objective. Noninvasive brain-computer interfaces (BCIs) assist paralyzed patients by providing access to the world without requiring surgical intervention. Prior work has suggested that EEG motor imagery based BCI can benefit from increased decoding accuracy through the application of deep learning methods, such as convolutional neural networks (CNNs).Approach. Here, we examine whether these improvements can generalize to practical scenarios such as continuous control tasks (as opposed to prior work reporting one classification per trial), whether valuable information remains latent outside of the motor cortex (as no prior work has compared full scalp coverage to motor only electrode montages), and the existing challenges to the practical implementation of deep-learning based continuous BCI control.Main results. We report that: (1) deep learning methods significantly increase offline performance compared to standard methods on an independent, large, and longitudinal online motor imagery BCI dataset with up to 4-classes and continuous 2D feedback; (2) our results suggest that a variety of neural biomarkers for BCI, including those outside the motor cortex, can be detected and used to improve performance through deep learning methods, and (3) tuning neural network output will be an important step in optimizing online BCI control, as we found the CNN models trained with full scalp EEG also significantly reduce the average trial length in a simulated online cursor control environment.Significance. This work demonstrates the benefits of CNNs classification during BCI control while providing evidence that electrode montage selection and the mapping of CNN output to device control will be important design choices in CNN based BCIs.
Collapse
Affiliation(s)
- James R Stieger
- Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, United States of America.,University of Minnesota, Minneapolis, MN, United States of America
| | - Stephen A Engel
- Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, United States of America.,University of Minnesota, Minneapolis, MN, United States of America
| | - Daniel Suma
- Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, United States of America
| | - Bin He
- Carnegie Mellon University, 5000 Forbes Avenue, Pittsburgh, PA 15213, United States of America
| |
Collapse
|
44
|
Lee M, Jeong JH, Kim YH, Lee SW. Decoding Finger Tapping With the Affected Hand in Chronic Stroke Patients During Motor Imagery and Execution. IEEE Trans Neural Syst Rehabil Eng 2021; 29:1099-1109. [PMID: 34101595 DOI: 10.1109/tnsre.2021.3087506] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
In stroke rehabilitation, motor imagery based on a brain-computer interface is an extremely useful method to control an external device and utilize neurofeedback. Many studies have reported on the classification performance of motor imagery to decode individual fingers in stroke patients compared with healthy controls. However, classification performance for a given limb is still low because the differences between patients owing to brain reorganization after stroke are not considered. We used electroencephalography signals from eleven healthy controls and eleven stroke patients in this study. The subjects performed a finger tapping task during motor execution, and motor imagery was performed with the dominant and affected hands in the healthy controls and stroke patients, respectively. All fingers except for the thumb were classified using the proposed framework based on a voting module. The averaged four-class accuracies during motor execution and motor imagery were 53.16 ± 8.42% and 46.94 ± 5.99% for the healthy controls and 53.17 ± 14.09% and 66.00 ± 14.96% for the stroke patients, respectively. Importantly, the classification accuracies in the stroke patients were statistically higher than those in healthy controls during motor imagery. However, there was no significant difference between the accuracies of motor execution and motor imagery. These findings show the potential for high classification performance for a given limb during motor imagery in stroke patients. These results can also provide insights into controlling an external device on the basis of a brain-computer interface.
Collapse
|
45
|
Ge Y, Zhang J. Robotic arm dynamics modelling and robust control based on model recognition method. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2021. [DOI: 10.3233/jifs-219069] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/15/2022]
Abstract
This paper analyzes the dynamics modelling and robust control of the robotic arm by using a model-based defines method. Firstly, the motion coupling relationship between the front and rear joints of the robotic arm is analyzed, and two kinds of motion decoupling modules based on planetary gear and pulley system are proposed, and the decoupling principle of the motion decoupling module is analyzed to realize the mechanical decoupling of the joint motion of the robotic arm. After that, a comprehensive test bench of two-degree-of-freedom robotic arm joint motion is constructed, and the factors influencing the decoupling effect of the mechanical decoupling module are analyzed through experiments to verify the effectiveness of the motion decoupling module. At the same time, the analysis also shows that: with the increase of the number of robotic arm joints, the number and volume of required decoupling modules increase, and the application of decoupling modules will significantly increase the volume, weight, and torque loss of the robotic arm, thus leading to the robotic arm’s large load to weight ratio which is not an advantage, therefore, mechanical decoupling is not suitable for robotic arms with more than 3 degrees of freedom. The design of a fuzzy incremental controller based on the model dialectic method is proposed for application in parallel robot control; it has universal approximation characteristics and can self-organize the velocity and position information of the parallel robot legs, and dynamically adjust the output of the controller by the designed affiliation function and control rules.
Collapse
Affiliation(s)
- Yi Ge
- Hunan Industry Polytechnic, Changsha Hunan, China
| | - Jing Zhang
- Hunan Industry Polytechnic, Changsha Hunan, China
| |
Collapse
|
46
|
Li MA, Ruan ZW. A novel decoding method for motor imagery tasks with 4D data representation and 3D convolutional neural networks. J Neural Eng 2021; 18. [PMID: 33836516 DOI: 10.1088/1741-2552/abf68b] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2020] [Accepted: 04/09/2021] [Indexed: 12/19/2022]
Abstract
Objective. Motor imagery electroencephalography (MI-EEG) produces one of the most commonly used biosignals in intelligent rehabilitation systems. The newly developed 3D convolutional neural network (3DCNN) is gaining increasing attention for its ability to recognize MI tasks. The key to successful identification of movement intention is dependent on whether the data representation can faithfully reflect the cortical activity induced by MI. However, the present data representation, which is often generated from partial source signals with time-frequency analysis, contains incomplete information. Therefore, it would be beneficial to explore a new type of data representation using raw spatiotemporal dipole information as well as the possible development of a matching 3DCNN.Approach.Based on EEG source imaging and 3DCNN, a novel decoding method for identifying MI tasks is proposed, called ESICNND. MI-EEG is mapped to the cerebral cortex by the standardized low resolution electromagnetic tomography algorithm, and the optimal sampling points of the dipoles are selected as the time of interest to best reveal the difference between any two MI tasks. Then, the initial subject coordinate system is converted to a magnetic resonance imaging coordinate system, followed by dipole interpolation and volume down-sampling; the resulting 3D dipole amplitude matrices are merged at the selected sampling points to obtain 4D dipole feature matrices (4DDFMs). These matrices are augmented by sliding window technology and input into a 3DCNN with a cascading architecture of three modules (3M3DCNN) to perform the extraction and classification of comprehensive features.Main results.Experiments are carried out on two public datasets; the average ten-fold CV classification accuracies reach 88.73% and 96.25%, respectively, and the statistical analysis demonstrates outstanding consistency and stability.Significance.The 4DDFMs reveals the variation of cortical activation in a 3D spatial cube with a temporal dimension and matches the 3M3DCNN well, making full use of the high-resolution spatiotemporal information from all dipoles.
Collapse
Affiliation(s)
- Ming-Ai Li
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, People's Republic of China.,Beijing Key Laboratory of Computational Intelligence and Intelligent System, Beijing 100124, People's Republic of China.,Engineering Research Center of Digital Community, Ministry of Education, Beijing 100124, People's Republic of China
| | - Zi-Wei Ruan
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, People's Republic of China
| |
Collapse
|
47
|
Sosnik R, Li Z. Reconstruction of hand, elbow and shoulder actual and imagined trajectories in 3D space using EEG current source dipoles. J Neural Eng 2021; 18. [PMID: 33752186 DOI: 10.1088/1741-2552/abf0d7] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/14/2020] [Accepted: 03/22/2021] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Growing evidence suggests that EEG electrode (sensor) potential time series (PTS) of slow cortical potentials (SCPs) hold motor neural correlates that can be used for motion trajectory prediction (MTP), commonly by multiple linear regression (mLR). It is not yet known whether arm-joint trajectories can be reliably decoded from current sources, computed from sensor data, from which brain areas they can be decoded and using which neural features. APPROACH In this study, the PTS of 44 sensors were fed into sLORETA source localization software to compute current source activity in 30 regions of interest (ROIs) found in a recent meta-analysis to be engaged in action execution, motor imagery and motor preparation. The current sources PTS and band-power time series (BTS) in several frequency bands and time lags were used to predict actual and imagined trajectories in 3D space of the three velocity components of the hand, elbow and shoulder of nine subjects using an mLR model. MAIN RESULTS For all arm joints and movement types, current source SCPs PTS contributed most to trajectory reconstruction with time lags 150ms, 116ms and 84ms providing the highest contribution, and current source BTS in any of the tested frequency bands was not informative. Person's correlation coefficient (r) averaged across movement types, arm joints and velocity components using source data was slightly lower than using sensor data (r=0.25 and r=0.28, respectively). For each ROI, the three current source dipoles had different contribution to the reconstruction of each of the three velocity components. SIGNIFICANCE Overall, our results demonstrate the feasibility of predicting of actual and imagined 3D trajectories of all arm joints from current sources, computed from scalp EEG. These findings may be used by developers of a future BCI as a validated set of contributing ROIs.
Collapse
Affiliation(s)
- Ronen Sosnik
- Electrical, Electronics and Communication Engineering, Holon Institute of Technology, 52 Golomb St., Holon, 5810201, ISRAEL
| | - Zheng Li
- State Key Laboratory of Cognitive Neuroscience and Learning & IDG/McGovern Institute for Brain Research, Beijing Normal University, Yingdong Building, Xinjiekouwai Street 19, Beijing Haidian, Beijing, 100875, CHINA
| |
Collapse
|
48
|
Zhang C, Kim YK, Eskandarian A. EEG-inception: an accurate and robust end-to-end neural network for EEG-based motor imagery classification. J Neural Eng 2021; 18. [PMID: 33691299 DOI: 10.1088/1741-2552/abed81] [Citation(s) in RCA: 38] [Impact Index Per Article: 12.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/26/2020] [Accepted: 03/10/2021] [Indexed: 12/14/2022]
Abstract
Classification of EEG-based motor imagery (MI) is a crucial non-invasive application in brain-computer interface (BCI) research. This paper proposes a novel convolutional neural network (CNN) architecture for accurate and robust EEG-based MI classification that outperforms the state-of-the-art methods. The proposed CNN model, namely EEG-Inception, is built on the backbone of the Inception-Time network, which has showed to be highly efficient and accurate for time-series classification. Also, the proposed network is an end-to-end classification, as it takes the raw EEG signals as the input and does not require complex EEG signal-preprocessing. Furthermore, this paper proposes a novel data augmentation method for EEG signals to enhance the accuracy, at least by 3%, and reduce overfitting with limited BCI datasets. The proposed model outperforms all state-of-the-art methods by achieving the average accuracy of 88.4% and 88.6% on the 2008 BCI Competition IV 2a (four-classes) and 2b datasets (binary-classes), respectively. Furthermore, it takes less than 0.025 seconds to test a sample suitable for real-time processing. Moreover, the classification standard deviation for nine different subjects achieves the lowest value of 5.5 for the 2b dataset and 7.1 for the 2a dataset, which validates that the proposed method is highly robust. From the experiment results, it can be inferred that the EEG-Inception network exhibits a strong potential as a subject-independent classifier for EEG-based MI tasks.
Collapse
Affiliation(s)
- Ce Zhang
- Mechanical Engineering, Virginia Polytechnic Institute and State University, 635 Prices Fork Rd, Blacksburg, Virginia, 24061-0131, UNITED STATES
| | - Young-Keun Kim
- Mechanical and Control Eng. Dept, Handong Global University, 558 Handong-ro Buk-gu, Pohang Gyeongbuk, Pohang, 37554, Korea (the Republic of)
| | - Azim Eskandarian
- Mechanical Engineering, Virginia Polytechnic Institute and State University, 635 Prices Fork Rd, Blacksburg, Virginia, 24060, UNITED STATES
| |
Collapse
|
49
|
Hong X, Zheng Q, Liu L, Chen P, Ma K, Gao Z, Zheng Y. Dynamic Joint Domain Adaptation Network for Motor Imagery Classification. IEEE Trans Neural Syst Rehabil Eng 2021; 29:556-565. [PMID: 33587702 DOI: 10.1109/tnsre.2021.3059166] [Citation(s) in RCA: 22] [Impact Index Per Article: 7.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
Electroencephalogram (EEG) has been widely used in brain computer interface (BCI) due to its convenience and reliability. The EEG-based BCI applications are majorly limited by the time-consuming calibration procedure for discriminative feature representation and classification. Existing EEG classification methods either heavily depend on the handcrafted features or require adequate annotated samples at each session for calibration. To address these issues, we propose a novel dynamic joint domain adaptation network based on adversarial learning strategy to learn domain-invariant feature representation, and thus improve EEG classification performance in the target domain by leveraging useful information from the source session. Specifically, we explore the global discriminator to align the marginal distribution across domains, and the local discriminator to reduce the conditional distribution discrepancy between sub-domains via conditioning on deep representation as well as the predicted labels from the classifier. In addition, we further investigate a dynamic adversarial factor to adaptively estimate the relative importance of alignment between the marginal and conditional distributions. To evaluate the efficacy of our method, extensive experiments are conducted on two public EEG datasets, namely, Datasets IIa and IIb of BCI Competition IV. The experimental results demonstrate that the proposed method achieves superior performance compared with the state-of-the-art methods.
Collapse
|
50
|
Development of a Human-Display Interface with Vibrotactile Feedback for Real-World Assistive Applications. SENSORS 2021; 21:s21020592. [PMID: 33467611 PMCID: PMC7830928 DOI: 10.3390/s21020592] [Citation(s) in RCA: 6] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 12/07/2020] [Revised: 01/11/2021] [Accepted: 01/11/2021] [Indexed: 12/21/2022]
Abstract
It is important to operate devices with control panels and touch screens assisted by haptic feedback in mobile environments such as driving automobiles and electric power wheelchairs. A lot of consideration is needed to give accurate haptic feedback, especially, presenting clear touch feedback to the elderly and people with reduced sensation is a very critical issue from healthcare and safety perspectives. In this study, we aimed to identify the perceptual characteristics for the frequency and direction of haptic vibration on the touch screen with vehicle-driving vibration and to propose an efficient haptic system based on these characteristics. As a result, we demonstrated that the detection threshold shift decreased at frequencies above 210 Hz due to the contact pressure during active touch, but the detection threshold shift increased at below 210 Hz. We found that the detection thresholds were 0.30–0.45 gpeak with similar sensitivity in the 80–270 Hz range. The haptic system implemented by reflecting the experimental results achieved characteristics suitable for use scenarios in automobiles. Ultimately, it could provide practical guidelines for the development of touch screens to give accurate touch feedback in the real-world environment.
Collapse
|