1
|
Lei Y, Wang D, Wang W, Qu H, Wang J, Shi B. Improving single-hand open/close motor imagery classification by error-related potentials correction. Heliyon 2023; 9:e18452. [PMID: 37520987 PMCID: PMC10382287 DOI: 10.1016/j.heliyon.2023.e18452] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/14/2022] [Revised: 07/15/2023] [Accepted: 07/18/2023] [Indexed: 08/01/2023] Open
Abstract
Objective The ability of a brain-computer interface (BCI) to classify brain activity in electroencephalograms (EEG) during motor imagery (MI) tasks is an important performance indicator. Because the cortical regions that drive the single-handed open and closed tasks overlap, it is difficult to classify the EEG signals during executing both tasks. Approach The addition of special EEG features can improve the accuracy of classifying single-hand open and closed tasks. In this work, we designed a hybrid BCI paradigm based on error-related potentials (ErrP) and motor imagery (MI) and proposed a strategy to correct the classification results of MI by using ErrP information. The ErrP and MI features of EEG data from 11 subjects were superimposed. Main results The corrected strategy improved the classification accuracy of single-hand open/close MI tasks from 52.3% to 73.7%, an increase of approximately 21%. Significance Our hybrid BCI paradigm improves the classification accuracy of single-hand MI by adding ErrP information, which provides a new approach for improving the classification performance of BCI.
Collapse
Affiliation(s)
- Yanghao Lei
- Institute of Robotics and Intelligent System, School of Mechanical Engineering, Xi'an Jiaotong University, Xi' an,710049, China
- Research Institute of NRR-Neurorehabilitation Robot, Xi' an Jiaotong University, Xi' an,710049, China
| | - Dong Wang
- Institute of Robotics and Intelligent System, School of Mechanical Engineering, Xi'an Jiaotong University, Xi' an,710049, China
- Research Institute of NRR-Neurorehabilitation Robot, Xi' an Jiaotong University, Xi' an,710049, China
| | - Weizhen Wang
- Institute of Robotics and Intelligent System, School of Mechanical Engineering, Xi'an Jiaotong University, Xi' an,710049, China
- Research Institute of NRR-Neurorehabilitation Robot, Xi' an Jiaotong University, Xi' an,710049, China
| | - Hao Qu
- Institute of Robotics and Intelligent System, School of Mechanical Engineering, Xi'an Jiaotong University, Xi' an,710049, China
- Research Institute of NRR-Neurorehabilitation Robot, Xi' an Jiaotong University, Xi' an,710049, China
| | - Jing Wang
- Institute of Robotics and Intelligent System, School of Mechanical Engineering, Xi'an Jiaotong University, Xi' an,710049, China
- Research Institute of NRR-Neurorehabilitation Robot, Xi' an Jiaotong University, Xi' an,710049, China
| | - Bin Shi
- PLA Rocket Force University of Engineering Xi'an, Xi' an, 710025, China
| |
Collapse
|
2
|
Sabaghypour S, Moghaddam HS, Farkhondeh Tale Navi F, Nazari MA, Soltanlou M. Do numbers make us handy? Behavioral and electrophysiological evidence for number-hand congruency effect. Acta Psychol (Amst) 2023; 233:103841. [PMID: 36709688 DOI: 10.1016/j.actpsy.2023.103841] [Citation(s) in RCA: 2] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/22/2021] [Revised: 08/26/2022] [Accepted: 01/19/2023] [Indexed: 01/30/2023] Open
Abstract
Finger counting facilitates numerical representations and mathematical processing. The current study investigated the association between finger counting habits and number processing by employing behavioral and electrophysiological measures. We explored whether small and large numerical primes influence the recognition of embodied target hand stimuli. Twenty-four right-handed participants that were grouped into right-starters (n = 13) and left-starters (n = 11) for finger counting performed a hand recognition task that consisted of numerical magnitudes as prime and hand recognition as targets. Based on the finger counting habits, congruent (i.e., left-starters: small number/left hand or large number/right hand; right-starters: small number/right hand or large number/left hand) and incongruent (i.e., left-starters: large number/left hand or small number/right hand; right-starters: large number/right hand or small number/left hand) conditions were presented to the participants. The participants were required to indicate whether the targets were left or right hand by simply pressing the left or the right key, respectively. Results indicated faster reaction times (RTs) for congruent as opposed to incongruent trials for all participants. The mean amplitude of the centro-parietal P300 component was significantly increased for the incongruent compared to congruent condition, indicating increased mental effort. Also, analysis of the latency of the P300 in terms of congruency effect in all participants revealed significant results. These combined results provide behavioral and electrophysiological evidence indicating the embodied nature of numbers. The results are interpreted in light of the general findings related to the P300 component. This research supports the association of number-hand representations and corroborates the idea of embodied numerosity.
Collapse
Affiliation(s)
- Saied Sabaghypour
- Department of Cognitive Neuroscience, Faculty of Education and Psychology, University of Tabriz, Tabriz, Iran.
| | - Hassan Sabouri Moghaddam
- Department of Cognitive Neuroscience, Faculty of Education and Psychology, University of Tabriz, Tabriz, Iran
| | - Farhad Farkhondeh Tale Navi
- Department of Cognitive Neuroscience, Faculty of Education and Psychology, University of Tabriz, Tabriz, Iran
| | - Mohammad Ali Nazari
- Department of Neuroscience, Faculty of Advanced Technologies in Medicine, Iran University of Medical Sciences, Tehran, Iran
| | - Mojtaba Soltanlou
- Department of Cognitive Neuroscience, Faculty of Education and Psychology, University of Tabriz, Tabriz, Iran; School of Psychology, University of Surrey, Guildford, UK
| |
Collapse
|
3
|
Zhang Z, Li D, Zhao Y, Fan Z, Xiang J, Wang X, Cui X. A flexible speller based on time-space frequency conversion SSVEP stimulation paradigm under dry electrode. Front Comput Neurosci 2023; 17:1101726. [PMID: 36817318 PMCID: PMC9929550 DOI: 10.3389/fncom.2023.1101726] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Accepted: 01/10/2023] [Indexed: 02/04/2023] Open
Abstract
Introduction Speller is the best way to express the performance of the brain-computer interface (BCI) paradigm. Due to its advantages of short analysis time and high accuracy, the SSVEP paradigm has been widely used in the BCI speller system based on the wet electrode. It is widely known that the wet electrode operation is cumbersome and that the subjects have a poor experience. In addition, in the asynchronous SSVEP system based on threshold analysis, the system flickers continuously from the beginning to the end of the experiment, which leads to visual fatigue. The dry electrode has a simple operation and provides a comfortable experience for subjects. The EOG signal can avoid the stimulation of SSVEP for a long time, thus reducing fatigue. Methods This study first designed the brain-controlled switch based on continuous blinking EOG signal and SSVEP signal to improve the flexibility of the BCI speller. Second, in order to increase the number of speller instructions, we designed the time-space frequency conversion (TSFC) SSVEP stimulus paradigm by constantly changing the time and space frequency of SSVEP sub-stimulus blocks, and designed a speller in a dry electrode environment. Results Seven subjects participated and completed the experiments. The results showed that the accuracy of the brain-controlled switch designed in this study was up to 94.64%, and all the subjects could use the speller flexibly. The designed 60-character speller based on the TSFC-SSVEP stimulus paradigm has an accuracy rate of 90.18% and an information transmission rate (ITR) of 117.05 bits/min. All subjects can output the specified characters in a short time. Discussion This study designed and implemented a multi-instruction SSVEP speller based on dry electrode. Through the combination of EOG and SSVEP signals, the speller can be flexibly controlled. The frequency of SSVEP stimulation sub-block is recoded in time and space by TSFC-SSVEP stimulation paradigm, which greatly improves the number of output instructions of BCI system in dry electrode environment. This work only uses FBCCA algorithm to test the stimulus paradigm, which requires a long stimulus time. In the future, we will use trained algorithms to study stimulus paradigm to improve its overall performance.
Collapse
|
4
|
Pan J, Chen X, Ban N, He J, Chen J, Huang H. Advances in P300 brain-computer interface spellers: toward paradigm design and performance evaluation. Front Hum Neurosci 2022; 16:1077717. [PMID: 36618996 PMCID: PMC9810759 DOI: 10.3389/fnhum.2022.1077717] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2022] [Accepted: 11/23/2022] [Indexed: 12/24/2022] Open
Abstract
A brain-computer interface (BCI) is a non-muscular communication technology that provides an information exchange channel for our brains and external devices. During the decades, BCI has made noticeable progress and has been applied in many fields. One of the most traditional BCI applications is the BCI speller. This article primarily discusses the progress of research into P300 BCI spellers and reviews four types of P300 spellers: single-modal P300 spellers, P300 spellers based on multiple brain patterns, P300 spellers with multisensory stimuli, and P300 spellers with multiple intelligent techniques. For each type of P300 speller, we further review several representative P300 spellers, including their design principles, paradigms, algorithms, experimental performance, and corresponding advantages. We particularly emphasized the paradigm design ideas, including the overall layout, individual symbol shapes and stimulus forms. Furthermore, several important issues and research guidance for the P300 speller were identified. We hope that this review can assist researchers in learning the new ideas of these novel P300 spellers and enhance their practical application capability.
Collapse
Affiliation(s)
- Jiahui Pan
- *Correspondence: Jiahui Pan Haiyun Huang
| | | | | | | | | | | |
Collapse
|
5
|
Li P, Su J, Belkacem AN, Cheng L, Chen C. Steady-state visually evoked potential collaborative BCI system deep learning classification algorithm based on multi-person feature fusion transfer learning-based convolutional neural network. Front Neurosci 2022; 16:971039. [PMID: 35958998 PMCID: PMC9360603 DOI: 10.3389/fnins.2022.971039] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/16/2022] [Accepted: 07/07/2022] [Indexed: 11/13/2022] Open
Abstract
ObjectiveThe conventional single-person brain–computer interface (BCI) systems have some intrinsic deficiencies such as low signal-to-noise ratio, distinct individual differences, and volatile experimental effect. To solve these problems, a centralized steady-state visually evoked potential collaborative BCI system (SSVEP-cBCI), which characterizes multi-person electroencephalography (EEG) feature fusion was constructed in this paper. Furthermore, three different feature fusion methods compatible with this new system were developed and applied to EEG classification, and a comparative analysis of their classification accuracy was performed with transfer learning-based convolutional neural network (TL-CNN) approach.ApproachAn EEG-based SSVEP-cBCI system was set up to merge different individuals’ EEG features stimulated by the instructions for the same task, and three feature fusion methods were adopted, namely parallel connection, serial connection, and multi-person averaging. The fused features were then input into CNN for classification. Additionally, transfer learning (TL) was applied first to a Tsinghua University (THU) benchmark dataset, and then to a collected dataset, so as to meet the CNN training requirement with a much smaller size of collected dataset and increase the classification accuracy. Ten subjects were recruited for data collection, and both datasets were used to gauge the three fusion algorithms’ performance.Main resultsThe results predicted by TL-CNN approach in single-person mode and in multi-person mode with the three feature fusion methods were compared. The experimental results show that each multi-person mode is superior to single-person mode. Within the 3 s time window, the classification accuracy of the single-person CNN is only 90.6%, while the same measure of the two-person parallel connection fusion method can reach 96.6%, achieving better classification effect.SignificanceThe results show that the three multi-person feature fusion methods and the deep learning classification algorithm based on TL-CNN can effectively improve the SSVEP-cBCI classification performance. The feature fusion method of multi -person parallel feature connection achieves better classification results. Different feature fusion methods can be selected in different application scenarios to further optimize cBCI.
Collapse
Affiliation(s)
- Penghai Li
- School of Integrated Circuit Science and Engineering, Tianjin University of Technology, Tianjin, China
| | - Jianxian Su
- School of Integrated Circuit Science and Engineering, Tianjin University of Technology, Tianjin, China
| | - Abdelkader Nasreddine Belkacem
- Department of Computer and Network Engineering, College of Information Technology, UAE University, Al Ain, United Arab Emirates
| | - Longlong Cheng
- China Electronics Cloud Brain Technology Co., Ltd., Tianjin, China
- *Correspondence: Longlong Cheng,
| | - Chao Chen
- Key Laboratory of Complex System Control Theory and Application, Tianjin University of Technology, Tianjin, China
- Chao Chen,
| |
Collapse
|
6
|
Li Q, Wu Y, Song Y, Zhao D, Sun M, Zhang Z, Wu J. A P300-Detection Method Based on Logistic Regression and a Convolutional Neural Network. Front Comput Neurosci 2022; 16:909553. [PMID: 35782086 PMCID: PMC9243506 DOI: 10.3389/fncom.2022.909553] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/31/2022] [Accepted: 05/13/2022] [Indexed: 11/29/2022] Open
Abstract
Background Electroencephalogram (EEG)-based brain-computer interface (BCI) systems are widely utilized in various fields, including health care, intelligent assistance, identity recognition, emotion recognition, and fatigue detection. P300, the main event-related potential, is the primary component detected by EEG-based BCI systems. Existing algorithms for P300 classification in EEG data usually perform well when tested in a single participant, although they exhibit significant decreases in accuracy when tested in new participants. We attempted to address this lack of generalizability associated with existing classification methods using a novel convolutional neural network (CNN) model developed using logistic regression (LR). Materials and Methods We proposed an LR-CNN model comprising two parts: a combined LR-based memory model and a CNN-based generalization model. The LR-based memory model can learn the individual features of participants and addresses the decrease in accuracy caused by individual differences when applied to new participants. The CNN-based generalization model can learn the common features among participants, thereby reducing overall classification bias and improving overall classification accuracy. Results We compared our method with existing, commonly used classification methods through three different sets of experiments. The experimental results indicated that our method could learn individual differences among participants. Compared with other commonly used classification methods, our method yielded a marked improvement (>90%) in classification among new participants. Conclusion The accuracy of the proposed model in the face of new participants is better than that of existing, commonly used classification methods. Such improvements in cross-subject test accuracy will aid in the development of BCI systems.
Collapse
Affiliation(s)
- Qi Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
- Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, China
- *Correspondence: Qi Li,
| | - Yan Wu
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Yu Song
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Di Zhao
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Meiqi Sun
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Zhilin Zhang
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
- Zhilin Zhang,
| | - Jinglong Wu
- Research Center for Medical Artificial Intelligence, Shenzhen Institute of Advanced Technology, Chinese Academy of Sciences, Shenzhen, China
| |
Collapse
|
7
|
Reconfiguration of Cortical Brain Network from Searching to Spotting for Dynamic Visual Targets. J Neurosci Methods 2022; 375:109577. [PMID: 35339507 DOI: 10.1016/j.jneumeth.2022.109577] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/11/2021] [Revised: 12/28/2021] [Accepted: 03/20/2022] [Indexed: 11/22/2022]
Abstract
BACKGROUND Detecting dynamic targets from complex visual scenes is an important problem in real world. However, the cognitive mechanism accounting for dynamic visual target detection remains unclear. NEW METHOD Herein, we aim to explore the cognitive process of dynamic visual target detection from searching to spotting and provide more concrete evidence for cognitive studies related to target detection. Cortical source responses with high spatiotemporal resolution were reconstructed from scalp EEG signals. Then, time-varying cortical networks were built using adaptive directed transfer function to explore the cognitive processes while detecting the dynamic visual target. RESULTS The experimental results demonstrated that the dynamic visual target detection enhanced the activation in both the visual and attention networks. Specially, the information flow from the middle occipital gyrus (MOG) mainly contributed to the position function, whereas the activation of the prefrontal cortex (PFC) reflected spatial attention maintenance. CONCLUSION The left "frontal-central-parietal" network played as a leading information source in dynamic target detection tasks. These findings provide new insights into cognitive processes of dynamic visual target detection. DATA AVAILABILITY STATEMENT The datasets in this study are available on request to the corresponding author.
Collapse
|
8
|
Wu Y, Zhou W, Lu Z, Li Q. A Spelling Paradigm With an Added Red Dot Improved the P300 Speller System Performance. Front Neuroinform 2020; 14:589169. [PMID: 33343323 PMCID: PMC7744603 DOI: 10.3389/fninf.2020.589169] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/30/2020] [Accepted: 11/02/2020] [Indexed: 11/13/2022] Open
Abstract
The traditional P300 speller system uses the flashing row or column spelling paradigm. However, the classification accuracy and information transfer rate of the P300 speller are not adequate for real-world application. To improve the performance of the P300 speller, we devised a new spelling paradigm in which the flashing row or column of a virtual character matrix is covered by a translucent green circle with a red dot in either the upper or lower half (GC-RD spelling paradigm). We compared the event-related potential (ERP) waveforms with a control paradigm (GC spelling paradigm), in which the flashing row or column of a virtual character matrix was covered by a translucent green circle only. Our experimental results showed that the amplitude of P3a at the parietal area and P3b at the frontal–central–parietal areas evoked by the GC-RD paradigm were significantly greater than those induced by the GC paradigm. Higher classification accuracy and information transmission rates were also obtained in the GC-RD system. Our results indicated that the added red dots increased attention and visuospatial information, resulting in an amplitude increase in both P3a and P3b, thereby improving the performance of the P300 speller system.
Collapse
Affiliation(s)
- Yan Wu
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Weiwei Zhou
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Zhaohua Lu
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Qi Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| |
Collapse
|
9
|
Lu Z, Li Q, Gao N, Yang J. Time-varying networks of ERPs in P300-speller paradigms based on spatially and semantically congruent audiovisual bimodality. J Neural Eng 2020; 17:046015. [DOI: 10.1088/1741-2552/aba07f] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/12/2022]
|
10
|
Saha S, Baumert M. Intra- and Inter-subject Variability in EEG-Based Sensorimotor Brain Computer Interface: A Review. Front Comput Neurosci 2020; 13:87. [PMID: 32038208 PMCID: PMC6985367 DOI: 10.3389/fncom.2019.00087] [Citation(s) in RCA: 78] [Impact Index Per Article: 19.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/21/2019] [Accepted: 12/16/2019] [Indexed: 12/05/2022] Open
Abstract
Brain computer interfaces (BCI) for the rehabilitation of motor impairments exploit sensorimotor rhythms (SMR) in the electroencephalogram (EEG). However, the neurophysiological processes underpinning the SMR often vary over time and across subjects. Inherent intra- and inter-subject variability causes covariate shift in data distributions that impede the transferability of model parameters amongst sessions/subjects. Transfer learning includes machine learning-based methods to compensate for inter-subject and inter-session (intra-subject) variability manifested in EEG-derived feature distributions as a covariate shift for BCI. Besides transfer learning approaches, recent studies have explored psychological and neurophysiological predictors as well as inter-subject associativity assessment, which may augment transfer learning in EEG-based BCI. Here, we highlight the importance of measuring inter-session/subject performance predictors for generalized BCI frameworks for both normal and motor-impaired people, reducing the necessity for tedious and annoying calibration sessions and BCI training.
Collapse
Affiliation(s)
- Simanto Saha
- School of Electrical and Electronic Engineering, The University of Adelaide, Adelaide, SA, Australia
| | - Mathias Baumert
- School of Electrical and Electronic Engineering, The University of Adelaide, Adelaide, SA, Australia
| |
Collapse
|
11
|
Lu Z, Li Q, Gao N, Yang J. The Self-Face Paradigm Improves the Performance of the P300-Speller System. Front Comput Neurosci 2020; 13:93. [PMID: 32009923 PMCID: PMC6974691 DOI: 10.3389/fncom.2019.00093] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2019] [Accepted: 12/20/2019] [Indexed: 12/14/2022] Open
Abstract
Objective: Previous studies have shown that the performance of the famous face P300-speller was better than that of the classical row/column flashing P300-speller. Furthermore, in some studies, the brain was more active when responding to one's own face than to a famous face, and a self-face stimulus elicited larger amplitude event-related potentials (ERPs) than did a famous face. Thus, we aimed to study the role of the self-face paradigm on further improving the performance of the P300-speller system with the famous face P300-speller paradigm as the control paradigm. Methods: We designed two facial P300-speller paradigms based on the self-face and a famous face (Ming Yao, a sports star; the famous face spelling paradigm) with a neutral expression. Results: ERP amplitudes were significantly greater in the self-face than in the famous face spelling paradigm at the parietal area from 340 to 480 ms (P300), from 480 to 600 ms (P600f), and at the fronto-central area from 700 to 800 ms. Offline and online classification results showed that the self-face spelling paradigm accuracies were significantly higher than those of the famous face spelling paradigm at superposing first two times (P < 0.05). Similar results were found for information transfer rates (P < 0.05). Conclusions: The self-face spelling paradigm significantly improved the performance of the P300-speller system. This has significant practical applications for brain-computer interfaces (BCIs) and could avoid infringement issues caused by using images of other people's faces.
Collapse
Affiliation(s)
- Zhaohua Lu
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Qi Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Ning Gao
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Jingjing Yang
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| |
Collapse
|
12
|
Lu Z, Li Q, Gao N, Yang J, Bai O. Happy emotion cognition of bimodal audiovisual stimuli optimizes the performance of the P300 speller. Brain Behav 2019; 9:e01479. [PMID: 31729840 PMCID: PMC6908870 DOI: 10.1002/brb3.1479] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/02/2019] [Revised: 09/17/2019] [Accepted: 10/26/2019] [Indexed: 12/18/2022] Open
Abstract
OBJECTIVE Prior studies of emotional cognition have found that emotion-based bimodal face and voice stimuli can elicit larger event-related potential (ERP) amplitudes and enhance neural responses compared with visual-only emotional face stimuli. Recent studies on brain-computer interface have shown that emotional face stimuli have significantly improved the performance of the traditional P300 speller system, but its performance needs to be further improved for practical applications. Therefore, we herein propose a novel audiovisual P300 speller based on bimodal emotional cognition to further improve the performance of the P300 system. METHODS The audiovisual P300 speller we proposed is based on happy emotions, with visual and auditory stimuli that consist of several pairs of smiling faces and audible chuckles (E-AV spelling paradigm) of different ages and sexes. The control paradigm was the visual-only emotional face P300 speller (E-V spelling paradigm). RESULTS We compared the ERP amplitudes, accuracy, and raw bit rate between the E-AV and E-V spelling paradigms. The target stimuli elicited significantly increased P300 amplitudes (p < .05) and P600 amplitudes (p < .05) in the E-AV spelling paradigm compared with those in the E-V paradigm. The E-AV spelling paradigm also significantly improved the spelling accuracy and the raw bit rate compared with those in the E-V paradigm at one superposition (p < .05) and at two superpositions (p < .05). SIGNIFICANCE The proposed emotion-based audiovisual spelling paradigm not only significantly improves the performance of the P300 speller, but also provides a basis for the development of various bimodal P300 speller systems, which is a step forward in the clinical application of brain-computer interfaces.
Collapse
Affiliation(s)
- Zhaohua Lu
- School of Computer Science and TechnologyChangchun University of Science and TechnologyChangchunChina
| | - Qi Li
- School of Computer Science and TechnologyChangchun University of Science and TechnologyChangchunChina
| | - Ning Gao
- School of Computer Science and TechnologyChangchun University of Science and TechnologyChangchunChina
| | - Jingjing Yang
- School of Computer Science and TechnologyChangchun University of Science and TechnologyChangchunChina
| | - Ou Bai
- Department of Electrical and Computer EngineeringFlorida International UniversityMiamiFLUSA
| |
Collapse
|
13
|
Lu Z, Li Q, Gao N, Yang J, Bai O. A Novel Audiovisual P300-Speller Paradigm Based on Cross-Modal Spatial and Semantic Congruence. Front Neurosci 2019; 13:1040. [PMID: 31611770 PMCID: PMC6777004 DOI: 10.3389/fnins.2019.01040] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/22/2019] [Accepted: 09/13/2019] [Indexed: 11/19/2022] Open
Abstract
Objective Although many studies have attempted to improve the performance of the visual-based P300-speller system, its performance is still not satisfactory. The current system has limitations for patients with neurodegenerative diseases, in which muscular control of the eyes may be impaired or deteriorate over time. Some studies have shown that the audiovisual stimuli with spatial and semantic congruence elicited larger event-related potential (ERP) amplitudes than do unimodal visual stimuli. Therefore, this study proposed a novel multisensory P300-speller based on audiovisual spatial and semantic congruence. Methods We designed a novel audiovisual P300-speller paradigm (AV spelling paradigm) in which the pronunciation and visual presentation of characters were matched in spatial position and semantics. We analyzed the ERP waveforms elicited in the AV spelling paradigm and visual-based spelling paradigm (V spelling paradigm) and compared the classification accuracies between these two paradigms. Results ERP analysis revealed significant differences in ERP amplitudes between the two paradigms in the following areas (AV > V): the frontal area at 60–140 ms, frontal–central–parietal area at 360–460 ms, frontal area at 700–800 ms, right temporal area at 380–480 and 700–780 ms, and left temporal area at 500–780 ms. Offline classification results showed that the accuracies were significantly higher in the AV spelling paradigm than in the V spelling paradigm after superposing 1, 2, 5, 6, 9, and 10 times (P < 0.05), and there were trends toward improvement in the accuracies at superposing 3, 4, 7, and 8 times (P = 0.06). Similar results were found for information transfer rate between V and AV spelling paradigms at 1, 2, 5, 6, and 10 superposition times (P < 0.05). Significance The proposed audiovisual P300-speller paradigm significantly improved the classification accuracies compared with the visual-based P300-speller paradigm. Our novel paradigm combines spatial and semantic features of two sensory modalities, and the present findings provide valuable insights into the development of multimodal ERP-based BCI paradigms.
Collapse
Affiliation(s)
- Zhaohua Lu
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Qi Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Ning Gao
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Jingjing Yang
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Ou Bai
- Department of Electrical and Computer Engineering, Florida International University, Miami, FL, United States
| |
Collapse
|