1
|
Kilani S, Aghili SN, Fathi Y, Sburlea AI. Optimization of transfer learning based on source sample selection in Euclidean space for P300-based brain-computer interfaces. Front Neurosci 2024; 18:1360709. [PMID: 39071181 PMCID: PMC11272559 DOI: 10.3389/fnins.2024.1360709] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/23/2023] [Accepted: 06/25/2024] [Indexed: 07/30/2024] Open
Abstract
Introduction Event-related potentials (ERPs), such as P300, are widely utilized for non-invasive monitoring of brain activity in brain-computer interfaces (BCIs) via electroencephalogram (EEG). However, the non-stationary nature of EEG signals and different data distributions among subjects create significant challenges for implementing real-time P300-based BCIs. This requires time-consuming calibration and a large number of training samples. Methods To address these challenges, this study proposes a transfer learning-based approach that uses a convolutional neural network for high-level feature extraction, followed by Euclidean space data alignment to ensure similar distributions of extracted features. Furthermore, a source selection technique based on the Euclidean distance metric was applied to measure the distance between each source feature sample and a reference point from the target domain. The samples with the lowest distance were then chosen to increase the similarity between source and target datasets. Finally, the transferred features are applied to a discriminative restricted Boltzmann machine classifier for P300 detection. Results The proposed method was evaluated on the state-of-the-art BCI Competition III dataset II and rapid serial visual presentation dataset. The results demonstrate that the proposed technique achieves an average accuracy of 97% for both online and offline after 15 repetitions, which is comparable to the state-of-the-art methods. Notably, the proposed approach requires <½ of the training samples needed by previous studies. Discussion Therefore, this technique offers an efficient solution for developing ERP-based BCIs with robust performance against reduced a number of training data.
Collapse
Affiliation(s)
- Sepideh Kilani
- Department of Biomedical Engineering, School of Electrical Engineering, Iran University of Science and Technology (IUST), Tehran, Iran
| | - Seyedeh Nadia Aghili
- Department of Biomedical Engineering, School of Electrical Engineering, Iran University of Science and Technology (IUST), Tehran, Iran
| | - Yaser Fathi
- Institute of Neuroscience, Universite Catholique de Louvain, Brussels, Belgium
| | - Andreea Ioana Sburlea
- Bernoulli Institute of Mathematics, Computer Science and Artificial Intelligence, Faculty of Science and Engineering, University of Groningen, Groningen, Netherlands
| |
Collapse
|
2
|
Portnova GV, Proskurnina EV. Acoustic and Subjective Basis of Emotional Perception in Comatose Patients: A Comparative Study. CLINICAL PSYCHOPHARMACOLOGY AND NEUROSCIENCE : THE OFFICIAL SCIENTIFIC JOURNAL OF THE KOREAN COLLEGE OF NEUROPSYCHOPHARMACOLOGY 2023; 21:701-714. [PMID: 37859443 PMCID: PMC10591159 DOI: 10.9758/cpn.22.1028] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 09/19/2022] [Revised: 02/15/2023] [Accepted: 03/12/2023] [Indexed: 10/21/2023]
Abstract
Objective : The acoustic stimulation in consciousness patients may improve the diagnosis assessment and the effectiveness of rehabilitation procedures. We aimed to investigate the event-related potential (ERP) response to emotional auditory stimuli in comatose patients. Methods : We measured the nonlinear and linear electroencephalogram (EEG) features, prepared the acoustic analysis of stimuli parameters, and assessed the subjective emotional rates of stimuli characteristics. Results : Patients with better outcomes had recognizable ERP responses and significant changes of the nonlinear EEG features to emotional sounds, unlike patients with worse outcomes. The response of comatose patients was attributed to acoustical features of emotional sounds, whereas the EEG response of healthy subjects was associated with their subjective feelings. The comatose patients demonstrated the variable EEG activity for neutral and emotional sounds. Conclusion : Thus, the EEG reactivity followed the better outcome of comatose patients to emotional stimuli. The study assumed the substantial differences of emotional stimuli perception in the healthy and unconscious brain.
Collapse
Affiliation(s)
- Galina V. Portnova
- Laboratory of Human Higher Nervous Activity, Institute of Higher Nervous Activity and Neurophysiology of the Russian Academy of Sciences, Moscow, Russia
- Department of Scientific Activities, Pushkin Institute of Russian Language, Moscow, Russia
| | - Elena V. Proskurnina
- Laboratory of Molecular Biology, Research Centre for Medical Genetics, Moscow, Russia
| |
Collapse
|
3
|
李 奇, 张 庭, 宋 雨, 刘 玉, 孙 美. [A design and evaluation of wearable p300 brain-computer interface system based on Hololens2]. SHENG WU YI XUE GONG CHENG XUE ZA ZHI = JOURNAL OF BIOMEDICAL ENGINEERING = SHENGWU YIXUE GONGCHENGXUE ZAZHI 2023; 40:709-717. [PMID: 37666761 PMCID: PMC10477399 DOI: 10.7507/1001-5515.202207055] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 07/21/2022] [Revised: 06/06/2023] [Indexed: 09/06/2023]
Abstract
Patients with amyotrophic lateral sclerosis ( ALS ) often have difficulty in expressing their intentions through language and behavior, which prevents them from communicating properly with the outside world and seriously affects their quality of life. The brain-computer interface (BCI) has received much attention as an aid for ALS patients to communicate with the outside world, but the heavy device causes inconvenience to patients in the application process. To improve the portability of the BCI system, this paper proposed a wearable P300-speller brain-computer interface system based on the augmented reality (MR-BCI). This system used Hololens2 augmented reality device to present the paradigm, an OpenBCI device to capture EEG signals, and Jetson Nano embedded computer to process the data. Meanwhile, to optimize the system's performance for character recognition, this paper proposed a convolutional neural network classification method with low computational complexity applied to the embedded system for real-time classification. The results showed that compared with the P300-speller brain-computer interface system based on the computer screen (CS-BCI), MR-BCI induced an increase in the amplitude of the P300 component, an increase in accuracy of 1.7% and 1.4% in offline and online experiments, respectively, and an increase in the information transfer rate of 0.7 bit/min. The MR-BCI proposed in this paper achieves a wearable BCI system based on guaranteed system performance. It has a positive effect on the realization of the clinical application of BCI.
Collapse
Affiliation(s)
- 奇 李
- 长春理工大学 计算机科学技术学院(长春 130022)School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, P. R. China
- 长春理工大学中山研究院(广东中山 528437)Zhongshan Institute of Changchun University of Science and Technology, Zhongshan, Guangdong 528437, P. R. China
| | - 庭嘉 张
- 长春理工大学 计算机科学技术学院(长春 130022)School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, P. R. China
| | - 雨 宋
- 长春理工大学 计算机科学技术学院(长春 130022)School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, P. R. China
| | - 玉龙 刘
- 长春理工大学 计算机科学技术学院(长春 130022)School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, P. R. China
| | - 美琪 孙
- 长春理工大学 计算机科学技术学院(长春 130022)School of Computer Science and Technology, Changchun University of Science and Technology, Changchun 130022, P. R. China
| |
Collapse
|
4
|
Gao C, Uchitomi H, Miyake Y. Influence of Multimodal Emotional Stimulations on Brain Activity: An Electroencephalographic Study. SENSORS (BASEL, SWITZERLAND) 2023; 23:4801. [PMID: 37430714 PMCID: PMC10221168 DOI: 10.3390/s23104801] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/20/2023] [Revised: 05/05/2023] [Accepted: 05/12/2023] [Indexed: 07/12/2023]
Abstract
This study aimed to reveal the influence of emotional valence and sensory modality on neural activity in response to multimodal emotional stimuli using scalp EEG. In this study, 20 healthy participants completed the emotional multimodal stimulation experiment for three stimulus modalities (audio, visual, and audio-visual), all of which are from the same video source with two emotional components (pleasure or unpleasure), and EEG data were collected using six experimental conditions and one resting state. We analyzed power spectral density (PSD) and event-related potential (ERP) components in response to multimodal emotional stimuli, for spectral and temporal analysis. PSD results showed that the single modality (audio only/visual only) emotional stimulation PSD differed from multi-modality (audio-visual) in a wide brain and band range due to the changes in modality and not from the changes in emotional degree. The most pronounced N200-to-P300 potential shifts occurred in monomodal rather than multimodal emotional stimulations. This study suggests that emotional saliency and sensory processing efficiency perform a significant role in shaping neural activity during multimodal emotional stimulation, with the sensory modality being more influential in PSD. These findings contribute to our understanding of the neural mechanisms involved in multimodal emotional stimulation.
Collapse
Affiliation(s)
- Chenguang Gao
- Department of Computer Science, Tokyo Institute of Technology, Yokohama 226-8502, Japan; (H.U.); (Y.M.)
| | | | | |
Collapse
|
5
|
Pan J, Chen X, Ban N, He J, Chen J, Huang H. Advances in P300 brain-computer interface spellers: toward paradigm design and performance evaluation. Front Hum Neurosci 2022; 16:1077717. [PMID: 36618996 PMCID: PMC9810759 DOI: 10.3389/fnhum.2022.1077717] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2022] [Accepted: 11/23/2022] [Indexed: 12/24/2022] Open
Abstract
A brain-computer interface (BCI) is a non-muscular communication technology that provides an information exchange channel for our brains and external devices. During the decades, BCI has made noticeable progress and has been applied in many fields. One of the most traditional BCI applications is the BCI speller. This article primarily discusses the progress of research into P300 BCI spellers and reviews four types of P300 spellers: single-modal P300 spellers, P300 spellers based on multiple brain patterns, P300 spellers with multisensory stimuli, and P300 spellers with multiple intelligent techniques. For each type of P300 speller, we further review several representative P300 spellers, including their design principles, paradigms, algorithms, experimental performance, and corresponding advantages. We particularly emphasized the paradigm design ideas, including the overall layout, individual symbol shapes and stimulus forms. Furthermore, several important issues and research guidance for the P300 speller were identified. We hope that this review can assist researchers in learning the new ideas of these novel P300 spellers and enhance their practical application capability.
Collapse
Affiliation(s)
- Jiahui Pan
- *Correspondence: Jiahui Pan Haiyun Huang
| | | | | | | | | | | |
Collapse
|
6
|
Kim M, Kim J, Heo D, Choi Y, Lee T, Kim SP. Effects of Emotional Stimulations on the Online Operation of a P300-Based Brain-Computer Interface. Front Hum Neurosci 2021; 15:612777. [PMID: 33767615 PMCID: PMC7987063 DOI: 10.3389/fnhum.2021.612777] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/30/2020] [Accepted: 02/05/2021] [Indexed: 11/23/2022] Open
Abstract
Using P300-based brain-computer interfaces (BCIs) in daily life should take into account the user's emotional state because various emotional conditions are likely to influence event-related potentials (ERPs) and consequently the performance of P300-based BCIs. This study aimed at investigating whether external emotional stimuli affect the performance of a P300-based BCI, particularly built for controlling home appliances. We presented a set of emotional auditory stimuli to subjects, which had been selected for each subject based on individual valence scores evaluated a priori, while they were controlling an electric light device using a P300-based BCI. There were four conditions regarding the auditory stimuli, including high valence, low valence, noise, and no sound. As a result, subjects controlled the electric light device using the BCI in real time with a mean accuracy of 88.14%. The overall accuracy and P300 features over most EEG channels did not show a significant difference between the four auditory conditions (p > 0.05). When we measured emotional states using frontal alpha asymmetry (FAA) and compared FAA across the auditory conditions, we also found no significant difference (p > 0.05). Our results suggest that there is no clear evidence to support a hypothesis that external emotional stimuli influence the P300-based BCI performance or the P300 features while people are controlling devices using the BCI in real time. This study may provide useful information for those who are concerned with the implementation of a P300-based BCI in practice.
Collapse
Affiliation(s)
| | | | | | | | | | - Sung-Phil Kim
- Department of Biomedical Engineering, Ulsan National Institute of Science and Technology, Ulsan, South Korea
| |
Collapse
|
7
|
Chailloux Peguero JD, Mendoza-Montoya O, Antelis JM. Single-Option P300-BCI Performance Is Affected by Visual Stimulation Conditions. SENSORS (BASEL, SWITZERLAND) 2020; 20:E7198. [PMID: 33339105 PMCID: PMC7765532 DOI: 10.3390/s20247198] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/06/2020] [Revised: 12/08/2020] [Accepted: 12/10/2020] [Indexed: 01/18/2023]
Abstract
The P300 paradigm is one of the most promising techniques for its robustness and reliability in Brain-Computer Interface (BCI) applications, but it is not exempt from shortcomings. The present work studied single-trial classification effectiveness in distinguishing between target and non-target responses considering two conditions of visual stimulation and the variation of the number of symbols presented to the user in a single-option visual frame. In addition, we also investigated the relationship between the classification results of target and non-target events when training and testing the machine-learning model with datasets containing different stimulation conditions and different number of symbols. To this end, we designed a P300 experimental protocol considering, as conditions of stimulation: the color highlighting or the superimposing of a cartoon face and from four to nine options. These experiments were carried out with 19 healthy subjects in 3 sessions. The results showed that the Event-Related Potentials (ERP) responses and the classification accuracy are stronger with cartoon faces as stimulus type and similar irrespective of the amount of options. In addition, the classification performance is reduced when using datasets with different type of stimulus, but it is similar when using datasets with different the number of symbols. These results have a special connotation for the design of systems, in which it is intended to elicit higher levels of evoked potentials and, at the same time, optimize training time.
Collapse
|
8
|
Li M, Yang G, Xu G. The Effect of the Graphic Structures of Humanoid Robot on N200 and P300 Potentials. IEEE Trans Neural Syst Rehabil Eng 2020; 28:1944-1954. [PMID: 32746323 DOI: 10.1109/tnsre.2020.3010250] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Humanoid robots are widely used in brain computer interface (BCI). Using a humanoid robot stimulus could increase the amplitude of event-related potentials (ERPs), which improves BCI performance. Since a humanoid robot contains many human elements, the element that increases the ERPs amplitude is unclear, and how to test the effect of it on the brain is a problem. This study used different graphic structures of an NAO humanoid robot to design three types of robot stimuli: a global robot, its local information, and its topological action. Ten subjects first conducted an odd-ball-based BCI (OD-BCI) by applying these stimuli. Then, they accomplished a delayed matching-to-sample task (DMST) that was used to specialize the encoding and retrieval phases of the OD-BCI task. In the retrieval phase of the DMST, the global stimulus induces the largest N200 and P300 potentials with the shortest latencies in the frontal, central, and occipital areas. This finding is in accordance with the P300 and classification performance of the OD-BCI task. When induced by the local stimulus, the subjects responded faster and more accurately in the retrieval phase of the DMST than in the other two conditions, indicating that the local stimulus improved the subject's responses. These results indicate that the OD-BCI task causes subject's retrieval work when the subject recognizes and outputs the stimulus. The global stimulus that contains topological and local elements could make brain react faster and induce larger ERPs, this finding could be used during the development of visual stimuli to improve BCI performance.
Collapse
|
9
|
Lu Z, Li Q, Gao N, Yang J. The Self-Face Paradigm Improves the Performance of the P300-Speller System. Front Comput Neurosci 2020; 13:93. [PMID: 32009923 PMCID: PMC6974691 DOI: 10.3389/fncom.2019.00093] [Citation(s) in RCA: 12] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/14/2019] [Accepted: 12/20/2019] [Indexed: 12/14/2022] Open
Abstract
Objective: Previous studies have shown that the performance of the famous face P300-speller was better than that of the classical row/column flashing P300-speller. Furthermore, in some studies, the brain was more active when responding to one's own face than to a famous face, and a self-face stimulus elicited larger amplitude event-related potentials (ERPs) than did a famous face. Thus, we aimed to study the role of the self-face paradigm on further improving the performance of the P300-speller system with the famous face P300-speller paradigm as the control paradigm. Methods: We designed two facial P300-speller paradigms based on the self-face and a famous face (Ming Yao, a sports star; the famous face spelling paradigm) with a neutral expression. Results: ERP amplitudes were significantly greater in the self-face than in the famous face spelling paradigm at the parietal area from 340 to 480 ms (P300), from 480 to 600 ms (P600f), and at the fronto-central area from 700 to 800 ms. Offline and online classification results showed that the self-face spelling paradigm accuracies were significantly higher than those of the famous face spelling paradigm at superposing first two times (P < 0.05). Similar results were found for information transfer rates (P < 0.05). Conclusions: The self-face spelling paradigm significantly improved the performance of the P300-speller system. This has significant practical applications for brain-computer interfaces (BCIs) and could avoid infringement issues caused by using images of other people's faces.
Collapse
Affiliation(s)
- Zhaohua Lu
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Qi Li
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Ning Gao
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| | - Jingjing Yang
- School of Computer Science and Technology, Changchun University of Science and Technology, Changchun, China
| |
Collapse
|
10
|
Lu Z, Li Q, Gao N, Yang J, Bai O. Happy emotion cognition of bimodal audiovisual stimuli optimizes the performance of the P300 speller. Brain Behav 2019; 9:e01479. [PMID: 31729840 PMCID: PMC6908870 DOI: 10.1002/brb3.1479] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.4] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Submit a Manuscript] [Subscribe] [Scholar Register] [Received: 07/02/2019] [Revised: 09/17/2019] [Accepted: 10/26/2019] [Indexed: 12/18/2022] Open
Abstract
OBJECTIVE Prior studies of emotional cognition have found that emotion-based bimodal face and voice stimuli can elicit larger event-related potential (ERP) amplitudes and enhance neural responses compared with visual-only emotional face stimuli. Recent studies on brain-computer interface have shown that emotional face stimuli have significantly improved the performance of the traditional P300 speller system, but its performance needs to be further improved for practical applications. Therefore, we herein propose a novel audiovisual P300 speller based on bimodal emotional cognition to further improve the performance of the P300 system. METHODS The audiovisual P300 speller we proposed is based on happy emotions, with visual and auditory stimuli that consist of several pairs of smiling faces and audible chuckles (E-AV spelling paradigm) of different ages and sexes. The control paradigm was the visual-only emotional face P300 speller (E-V spelling paradigm). RESULTS We compared the ERP amplitudes, accuracy, and raw bit rate between the E-AV and E-V spelling paradigms. The target stimuli elicited significantly increased P300 amplitudes (p < .05) and P600 amplitudes (p < .05) in the E-AV spelling paradigm compared with those in the E-V paradigm. The E-AV spelling paradigm also significantly improved the spelling accuracy and the raw bit rate compared with those in the E-V paradigm at one superposition (p < .05) and at two superpositions (p < .05). SIGNIFICANCE The proposed emotion-based audiovisual spelling paradigm not only significantly improves the performance of the P300 speller, but also provides a basis for the development of various bimodal P300 speller systems, which is a step forward in the clinical application of brain-computer interfaces.
Collapse
Affiliation(s)
- Zhaohua Lu
- School of Computer Science and TechnologyChangchun University of Science and TechnologyChangchunChina
| | - Qi Li
- School of Computer Science and TechnologyChangchun University of Science and TechnologyChangchunChina
| | - Ning Gao
- School of Computer Science and TechnologyChangchun University of Science and TechnologyChangchunChina
| | - Jingjing Yang
- School of Computer Science and TechnologyChangchun University of Science and TechnologyChangchunChina
| | - Ou Bai
- Department of Electrical and Computer EngineeringFlorida International UniversityMiamiFLUSA
| |
Collapse
|