1
|
Niu L, Bin J, Wang JKS, Zhan G, Jia J, Zhang L, Gan Z, Kang X. Effect of 3D paradigm synchronous motion for SSVEP-based hybrid BCI-VR system. Med Biol Eng Comput 2023; 61:2481-2495. [PMID: 37191865 DOI: 10.1007/s11517-023-02845-8] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/25/2022] [Accepted: 05/05/2023] [Indexed: 05/17/2023]
Abstract
A brain-computer interface (BCI) system and virtual reality (VR) are integrated as a more interactive hybrid system (BCI-VR) that allows the user to manipulate the car. A virtual scene in the VR system that is the same as the physical environment is built, and the object's movement can be observed in the VR scene. The four-class three-dimensional (3D) paradigm is designed and moves synchronously in virtual reality. The dynamic paradigm may affect their attention according to the experimenters' feedback. Fifteen subjects in our experiment steered the car according to a specified motion trajectory. According to our online experimental result, different motion trajectories of the paradigm have various effects on the system's performance, and training can mitigate this adverse effect. Moreover, the hybrid system using frequencies between 5 and 10 Hz indicates better performance than those using lower or higher stimulation frequencies. The experiment results show a maximum average accuracy of 0.956 and a maximum information transfer rate (ITR) of 41.033 bits/min. It suggests that a hybrid system provides a high-performance way of brain-computer interaction. This research could encourage more interesting applications involving BCI and VR technologies.
Collapse
Affiliation(s)
- Lan Niu
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | - Jianxiong Bin
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | | | - Gege Zhan
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | - Jie Jia
- Department of Rehabilitation Medicine, Huashan Hospital, Fudan University, Shanghai, 200040, China
| | - Lihua Zhang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | - Zhongxue Gan
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China
| | - Xiaoyang Kang
- Laboratory for Neural Interface and Brain Computer Interface, Engineering Research Center of AI & Robotics, Shanghai Engineering Research Center of AI & Robotics, MOE Frontiers Center for Brain Science, State Key Laboratory of Medical Neurobiology, Institute of AI & Robotics, Institute of Meta-Medical, Academy for Engineering & Technology, Ministry of Education, FudanUniversity, Shanghai, China.
- Ji Hua Laboratory, Foshan, 528000, Guangdong Province, China.
- Yiwu Research Institute of Fudan University, Chengbei Road, Yiwu City, 322000, Zhejiang, China.
- Research Center for Intelligent Sensing, Zhejiang Lab, Hangzhou, 311100, China.
| |
Collapse
|
2
|
Mussi MG, Adams KD. EEG hybrid brain-computer interfaces: A scoping review applying an existing hybrid-BCI taxonomy and considerations for pediatric applications. Front Hum Neurosci 2022; 16:1007136. [DOI: 10.3389/fnhum.2022.1007136] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2022] [Accepted: 10/27/2022] [Indexed: 11/18/2022] Open
Abstract
Most hybrid brain-computer interfaces (hBCI) aim at improving the performance of single-input BCI. Many combinations are possible to configure an hBCI, such as using multiple brain input signals, different stimuli or more than one input system. Multiple studies have been done since 2010 where such interfaces have been tested and analyzed. Results and conclusions are promising but little has been discussed as to what is the best approach for the pediatric population, should they use hBCI as an assistive technology. Children might face greater challenges when using BCI and might benefit from less complex interfaces. Hence, in this scoping review we included 42 papers that developed hBCI systems for the purpose of control of assistive devices or communication software, and we analyzed them through the lenses of potential use in clinical settings and for children. We extracted taxonomic categories proposed in previous studies to describe the types of interfaces that have been developed. We also proposed interface characteristics that could be observed in different hBCI, such as type of target, number of targets and number of steps before selection. Then, we discussed how each of the extracted characteristics could influence the overall complexity of the system and what might be the best options for applications for children. Effectiveness and efficiency were also collected and included in the analysis. We concluded that the least complex hBCI interfaces might involve having a brain inputs and an external input, with a sequential role of operation, and visual stimuli. Those interfaces might also use a minimal number of targets of the strobic type, with one or two steps before the final selection. We hope this review can be used as a guideline for future hBCI developments and as an incentive to the design of interfaces that can also serve children who have motor impairments.
Collapse
|
3
|
Yue Z, Wu Q, Ren SY, Li M, Shi B, Pan Y, Wang J. A novel multiple time-frequency sequential coding strategy for hybrid brain-computer interface. Front Hum Neurosci 2022; 16:859259. [PMID: 35966991 PMCID: PMC9372511 DOI: 10.3389/fnhum.2022.859259] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/21/2022] [Accepted: 06/28/2022] [Indexed: 11/18/2022] Open
Abstract
Background For brain-computer interface (BCI) communication, electroencephalography provides a preferable choice due to its high temporal resolution and portability over other neural recording techniques. However, current BCIs are unable to sufficiently use the information from time and frequency domains simultaneously. Thus, we proposed a novel hybrid time-frequency paradigm to investigate better ways of using the time and frequency information. Method We adopt multiple omitted stimulus potential (OSP) and steady-state motion visual evoked potential (SSMVEP) to design the hybrid paradigm. A series of pre-experiments were undertaken to study factors that would influence the feasibility of the hybrid paradigm and the interaction between multiple features. After that, a novel Multiple Time-Frequencies Sequential Coding (MTFSC) strategy was introduced and explored in experiments. Results Omissions with multiple short and long durations could effectively elicit time and frequency features, including the multi-OSP, ERP, and SSVEP in this hybrid paradigm. The MTFSC was feasible and efficient. The preliminary online analysis showed that the accuracy and the ITR of the nine-target stimulator over thirteen subjects were 89.04% and 36.37 bits/min. Significance This study first combined the SSMVEP and multi-OSP in a hybrid paradigm to produce robust and abundant time features for coding BCI. Meanwhile, the MTFSC proved feasible and showed great potential in improving performance, such as expanding the number of BCI targets by better using time information in specific stimulated frequencies. This study holds promise for designing better BCI systems with a novel coding method.
Collapse
Affiliation(s)
- Zan Yue
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
| | - Qiong Wu
- Beijing Tsinghua Changgeng Hospital, Tsinghua University, Beijing, China
| | - Shi-Yuan Ren
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
| | - Man Li
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
| | - Bin Shi
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
| | - Yu Pan
- Beijing Tsinghua Changgeng Hospital, Tsinghua University, Beijing, China
- *Correspondence: Yu Pan
| | - Jing Wang
- Institute of Robotics and Intelligent Systems, Xi'an Jiaotong University, Xi'an, China
- Jing Wang
| |
Collapse
|
4
|
Zhang D, Liu S, Wang K, Zhang J, Chen D, Zhang Y, Nie L, Yang J, Shinntarou F, Wu J, Yan T. Machine-vision fused brain machine interface based on dynamic augmented reality visual stimulation. J Neural Eng 2021; 18. [PMID: 34607320 DOI: 10.1088/1741-2552/ac2c9e] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/25/2021] [Accepted: 10/04/2021] [Indexed: 11/11/2022]
Abstract
Objective.Brain-machine interfaces (BMIs) interpret human intent into machine reactions, and the visual stimulation (VS) paradigm is one of the most widely used of these approaches. Although VS-based BMIs have a relatively high information transfer rate (ITR), it is still difficult for BMIs to control machines in dynamic environments (for example, grabbing a dynamic object or targeting a walking person).Approach.In this study, we utilized a BMI based on augmented reality (AR) VS (AR-VS). The proposed VS was dynamically generated based on machine vision, and human intent was interpreted by a dynamic decision time interval approach. A robot based on the coordination of a task and self-motion system was controlled by the proposed paradigm in a fast and flexible state.Methods.Objects in scenes were first recognized by machine vision and tracked by optical flow. AR-VS was generated based on the objects' parameters. The number and distribution of VS was confirmed by the recognized objects. Electroencephalogram (EEG) features corresponding to VS and human intent were collected by a dry-electrode EEG cap and determined by the filter bank canonical correlation analysis method. Key parameters in the AR-VS, including the effect of VS size, frequency, dynamic object moving speed, ITR and the performance of the BMI-controlled robot, were analyzed.Conclusion and significance.The ITR of the proposed AR-VS paradigm for nine healthy subjects was 36.3 ± 20.1 bits min-1. In the online robot control experiment, brain-controlled hybrid tasks including self-moving and grabbing objects were 64% faster than when using the traditional steady-state visual evoked potential paradigm. The proposed paradigm based on AR-VS could be optimized and adopted in other kinds of VS-based BMIs, such as P300, omitted stimulus potential, and miniature event-related potential paradigms, for better results in dynamic environments.
Collapse
Affiliation(s)
- Deyu Zhang
- Intelligent Robotics Institute, School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Siyu Liu
- School of Life Science, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Kai Wang
- School of Life Science, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Jian Zhang
- Intelligent Robotics Institute, School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Duanduan Chen
- School of Life Science, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Yilong Zhang
- School of Life Science, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Li Nie
- Intelligent Robotics Institute, School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Jiajia Yang
- Intelligent Robotics Institute, School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Funabashi Shinntarou
- Intelligent Robotics Institute, School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Jinglong Wu
- Intelligent Robotics Institute, School of Mechatronical Engineering, Beijing Institute of Technology, Beijing, People's Republic of China
| | - Tianyi Yan
- School of Life Science, Beijing Institute of Technology, Beijing, People's Republic of China
| |
Collapse
|
5
|
Hong J, Qin X. Signal processing algorithms for SSVEP-based brain computer interface: State-of-the-art and recent developments. JOURNAL OF INTELLIGENT & FUZZY SYSTEMS 2021. [DOI: 10.3233/jifs-201280] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/19/2023]
Abstract
Over past two decades, steady-state evoked potentials (SSVEP)-based brain computer interface (BCI) systems have been extensively developed. As we all know, signal processing algorithms play an important role in this BCI. However, there is no comprehensive review of the latest development of signal processing algorithms for SSVEP-based BCI. By analyzing the papers published in authoritative journals in nearly five years, signal processing algorithms of preprocessing, feature extraction and classification modules are discussed in detail. In addition, other aspects existed in this BCI are mentioned. The following key problems are solved. (1) In recent years, which signal processing algorithms are frequently used in each module? (2) Which signal processing algorithms attract more attention in recent years? (3) Which modules are the key to signal processing in BCI field? This information is very important for choosing the appropriate algorithms, and can also be considered as a reference for further research. Simultaneously, we hope that this work can provide relevant BCI researchers with valuable information about the latest trends of signal processing algorithms for SSVEP-based BCI systems.
Collapse
Affiliation(s)
- Jie Hong
- School of Mechanical Engineering, Northwestern Polytechnical University, Xi’an, Shaanxi, China
| | - Xiansheng Qin
- School of Mechanical Engineering, Northwestern Polytechnical University, Xi’an, Shaanxi, China
| |
Collapse
|
6
|
Katyal EA, Singla R. EEG-based hybrid QWERTY mental speller with high information transfer rate. Med Biol Eng Comput 2021; 59:633-661. [PMID: 33594631 DOI: 10.1007/s11517-020-02310-w] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/07/2020] [Accepted: 12/30/2020] [Indexed: 11/27/2022]
Abstract
BACKGROUND Brain-computer interface (BCI) spellers detect variations in brain waves to help subjects communicate with the world. This study introduces a P300-SSVEP hybrid BCI-based QWERTY speller. METHODS The proposed hybrid speller, combines SSVEP and P300 features using a hybrid paradigm. P300 was used as time division multiplexing index which results in the use of lesser number of assumed frequencies for SSVEP elicitation. Each flickering frequency was also assigned a unique colour, to enhance system accuracy. RESULTS On the basis of 20 subjects, an average accuracy of classification of 96.42% and a mean information transfer rate (ITR) of 131.0 bits per min. (BPM) was achieved during the free spelling trial (trial-F). COMPARISON The t test results revealed that the hybrid QWERTY speller performed significantly better (on the basis of mean classification accuracy and ITR) as compared to the traditional P300 speller) and the QWERTY SSVEP speller. Also, the amount of time taken to spell a word was significantly lesser in the case of hybrid QWERTY speller in contrast to traditional P300 speller while it was almost the same as compared to QWERTY SSVEP speller. CONCLUSION QWERTY speller outperformed the stereotypical P300 speller as well as QWERTY SSVEP speller.
Collapse
Affiliation(s)
- Er Akshay Katyal
- ICE Department, Dr B.R. Ambedkar N.I.T. Jalandhar, GT Road Bye-Pass, Jalandhar, Punjab, 144011, India.
| | - Rajesh Singla
- ICE Department, Dr B.R. Ambedkar N.I.T. Jalandhar, GT Road Bye-Pass, Jalandhar, Punjab, 144011, India
| |
Collapse
|
7
|
Katyal A, Singla R. Synchronized Detection of Evoked Potentials to Drive a High Information Transfer Rate Hybrid Brain-Computer Interface Application. ADVANCED BIOMEDICAL ENGINEERING 2021. [DOI: 10.14326/abe.10.58] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/04/2022] Open
Affiliation(s)
- Akshay Katyal
- Department of Instrumentation and Control Engineering, Dr BR Ambedkar National Institute of Technology Jalandhar
| | - Rajesh Singla
- Department of Instrumentation and Control Engineering, Dr BR Ambedkar National Institute of Technology Jalandhar
| |
Collapse
|
8
|
Katyal A, Singla R. A novel hybrid paradigm based on steady state visually evoked potential & P300 to enhance information transfer rate. Biomed Signal Process Control 2020. [DOI: 10.1016/j.bspc.2020.101884] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023]
|
9
|
Hernández OH. Circadian rhythms of omitted stimulus potentials in the crayfish brain. BIOL RHYTHM RES 2018. [DOI: 10.1080/09291016.2017.1336885] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
Affiliation(s)
- Oscar Hernando Hernández
- Centro de Investigaciones Biomédicas, Universidad Autónoma de Campeche, San Francisco de Campeche, Campeche, México
- Hospital General de Especialidades “Dr. Javier Buenfil Osorio”, San Francisco de Campeche, Campeche, México
| |
Collapse
|
10
|
Hong KS, Khan MJ. Hybrid Brain-Computer Interface Techniques for Improved Classification Accuracy and Increased Number of Commands: A Review. Front Neurorobot 2017; 11:35. [PMID: 28790910 PMCID: PMC5522881 DOI: 10.3389/fnbot.2017.00035] [Citation(s) in RCA: 116] [Impact Index Per Article: 16.6] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/24/2017] [Accepted: 07/03/2017] [Indexed: 12/11/2022] Open
Abstract
In this article, non-invasive hybrid brain-computer interface (hBCI) technologies for improving classification accuracy and increasing the number of commands are reviewed. Hybridization combining more than two modalities is a new trend in brain imaging and prosthesis control. Electroencephalography (EEG), due to its easy use and fast temporal resolution, is most widely utilized in combination with other brain/non-brain signal acquisition modalities, for instance, functional near infrared spectroscopy (fNIRS), electromyography (EMG), electrooculography (EOG), and eye tracker. Three main purposes of hybridization are to increase the number of control commands, improve classification accuracy and reduce the signal detection time. Currently, such combinations of EEG + fNIRS and EEG + EOG are most commonly employed. Four principal components (i.e., hardware, paradigm, classifiers, and features) relevant to accuracy improvement are discussed. In the case of brain signals, motor imagination/movement tasks are combined with cognitive tasks to increase active brain-computer interface (BCI) accuracy. Active and reactive tasks sometimes are combined: motor imagination with steady-state evoked visual potentials (SSVEP) and motor imagination with P300. In the case of reactive tasks, SSVEP is most widely combined with P300 to increase the number of commands. Passive BCIs, however, are rare. After discussing the hardware and strategies involved in the development of hBCI, the second part examines the approaches used to increase the number of control commands and to enhance classification accuracy. The future prospects and the extension of hBCI in real-time applications for daily life scenarios are provided.
Collapse
Affiliation(s)
- Keum-Shik Hong
- School of Mechanical Engineering, Pusan National University, Busan, South Korea.,Department of Cogno-Mechatronics Engineering, Pusan National University, Busan, South Korea
| | - Muhammad Jawad Khan
- School of Mechanical Engineering, Pusan National University, Busan, South Korea
| |
Collapse
|
11
|
Hernández OH, Hernández-Sánchez KM. Omitted Stimulus Potential depends on the sensory modality. Acta Neurobiol Exp (Wars) 2017. [DOI: 10.21307/ane-2017-062] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.1] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|