51
|
Miao Y, Chen S, Zhang X, Jin J, Xu R, Daly I, Jia J, Wang X, Cichocki A, Jung TP. BCI-Based Rehabilitation on the Stroke in Sequela Stage. Neural Plast 2020; 2020:8882764. [PMID: 33414824 PMCID: PMC7752268 DOI: 10.1155/2020/8882764] [Citation(s) in RCA: 27] [Impact Index Per Article: 6.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/28/2020] [Revised: 11/25/2020] [Accepted: 11/30/2020] [Indexed: 11/24/2022] Open
Abstract
Background Stroke is the leading cause of serious and long-term disability worldwide. Survivors may recover some motor functions after rehabilitation therapy. However, many stroke patients missed the best time period for recovery and entered into the sequela stage of chronic stroke. Method Studies have shown that motor imagery- (MI-) based brain-computer interface (BCI) has a positive effect on poststroke rehabilitation. This study used both virtual limbs and functional electrical stimulation (FES) as feedback to provide patients with a closed-loop sensorimotor integration for motor rehabilitation. An MI-based BCI system acquired, analyzed, and classified motor attempts from electroencephalogram (EEG) signals. The FES system would be activated if the BCI detected that the user was imagining wrist dorsiflexion on the instructed side of the body. Sixteen stroke patients in the sequela stage were randomly assigned to a BCI group and a control group. All of them participated in rehabilitation training for four weeks and were assessed by the Fugl-Meyer Assessment (FMA) of motor function. Results The average improvement score of the BCI group was 3.5, which was higher than that of the control group (0.9). The active EEG patterns of the four patients in the BCI group whose FMA scores increased gradually became centralized and shifted to sensorimotor areas and premotor areas throughout the study. Conclusions Study results showed evidence that patients in the BCI group achieved larger functional improvements than those in the control group and that the BCI-FES system is effective in restoring motor function to upper extremities in stroke patients. This study provides a more autonomous approach than traditional treatments used in stroke rehabilitation.
Collapse
|
52
|
Ko LW, D SVS, Huang Y, Lu YC, Shaw S, Jung TP. SSVEP-assisted RSVP Brain-Computer Interface paradigm for multi-target classification. J Neural Eng 2020; 18. [PMID: 33291083 DOI: 10.1088/1741-2552/abd1c0] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/25/2020] [Accepted: 12/08/2020] [Indexed: 11/12/2022]
Abstract
Brain-Computer Interface (BCI) is actively involved in optimizing the communication medium between the human brain and external devices. OBJECTIVE Rapid serial visual presentation (RSVP) is a robust and highly efficient BCI technique in recognizing target objects but suffers from limited target selections. The BCI systems that combine steady-state visual evoked potential (SSVEP) and RSVP can mitigate this limitation and allow users to operate on multiple targets. APPROACH This study proposes a novel SSVEP-assisted RSVP BCI model to improve the performance of classifying the target/non-target objects in a multi-target scenario. In this paradigm, SSVEP stimuli helps in identifying the user's focus location and RSVP stimuli that elicits event-related potentials (ERPs) differentiate target and non-target objects. MAIN RESULTS The proposed model achieved an offline accuracy of 81.59% by using 12 electroencephalogram (EEG) channels and an online (real-time) accuracy of 78.10% when only 4 EEG channels are considered. Further, the biomarkers of physiological states are analyzed to assess the cognitive states (mental fatigue and user attention) of the participants based on resting theta and alpha band powers. The results indicate an inverse relationship between the BCI performance and the resting EEG power, validating that the subjects' performance is affected by physiological states for prolonged BCI tasks. SIGNIFICANCE Our findings demonstrate that the combination of SSVEP and RSVP stimuli improves the BCI performance and further enhances the possibility of performing multiple user command tasks, which are inevitable in real-world applications. Additionally, the cognitive state biomarkers discussed imply the need for an efficient and attractive experimental paradigm that reduces the physiological state disparities and provide enhanced BCI performance.
Collapse
|
53
|
Chiang KJ, Wei CS, Nakanishi M, Jung TP. Boosting template-based SSVEP decoding by cross-domain transfer learning. J Neural Eng 2020; 18. [PMID: 33203813 DOI: 10.1088/1741-2552/abcb6e] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/18/2020] [Accepted: 11/16/2020] [Indexed: 11/12/2022]
Abstract
OBJECTIVE This study aims to establish a generalized transfer-learning framework for boosting the performance of steady-state visual evoked potential(SSVEP)-based brain-computer interfaces (BCIs) by leveraging cross-domain data transferring. APPROACH We enhanced the state-of-the-art template-based SSVEP decoding through incorporating a least-squares transformation (LST)-based transfer learning to leverage calibration data across multiple domains (sessions, subjects, and EEG montages). MAIN RESULTS Study results verified the efficacy of LST in obviating the variability of SSVEPs when transferring existing data across domains. Furthermore, the LST-based method achieved significantly higher SSVEP-decoding accuracy than the standard task-related component analysis (TRCA)-based method and the non-LST naive transfer-learning method. SIGNIFICANCE This study demonstrated the capability of the LST-based transfer learning to leverage existing data across subjects and/or devices with an in-depth investigation of its rationale and behavior in various circumstances. The proposed framework significantly improved the SSVEP decoding accuracy over the standard TRCA approach when calibration data are limited. Its performance in calibration reduction could facilitate plug-and-play SSVEP-based BCIs and further practical applications.
Collapse
|
54
|
Xu M, Han J, Wang Y, Jung TP, Ming D. Implementing Over 100 Command Codes for a High-Speed Hybrid Brain-Computer Interface Using Concurrent P300 and SSVEP Features. IEEE Trans Biomed Eng 2020; 67:3073-3082. [DOI: 10.1109/tbme.2020.2975614] [Citation(s) in RCA: 62] [Impact Index Per Article: 15.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
55
|
Mei J, Xu M, Wang L, Ke Y, Wang Y, Jung TP, Ming D. Using SSVEP-BCI to Continuous Control a Quadcopter with 4-DOF Motions .. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:4745-4748. [PMID: 33019051 DOI: 10.1109/embc44109.2020.9176131] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Brain-computer interfaces (BCIs) allow for translating electroencephalogram (EEG) into control commands, e.g., to control a quadcopter. This study, we developed a practical BCI based on steady-state visually evoked potential (SSVEP) for continuous control of a quadcopter from the first-person perspective. Users watched with the video stream from a camera on the quadcopter. An innovative user interface was developed by embedding 12 SSVEP flickers into the video stream, which corresponded to the flight commands of 'take-off,' 'land,' 'hover,' 'keep-going,' 'clockwise,' 'counter-clockwise' and rectilinear motions in six directions, respectively. The command was updated every 400ms by decoding the collected EEG data using a combined classification algorithm based on task-related component analysis (TRCA) and linear discriminant analysis (LDA). The quadcopter flew in the 3-D space according to the control vector that was determined by the latest four commands. Three novices participated in this study. They were asked to control the quadcopter by either the brain or hands to fly through a circle and land on the target zone. As a result, the time consumption ratio of brain-control to hand-control was as low as 1.34, which means the BCI performance was close to hands. The information transfer rate reached a peak of 401.79 bits/min in the simulated online experiment. These results demonstrate the proposed SSVEP-BCI system is efficient for controlling the quadcopter.
Collapse
|
56
|
Yue L, Xiao X, Xu M, Chen L, Wang Y, Jung TP, Ming D. A brain-computer interface based on high-frequency steady-state asymmetric visual evoked potentials .. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:3090-3093. [PMID: 33018658 DOI: 10.1109/embc44109.2020.9176855] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Steady State Visual Evoked Potentials (SSVEPs) have been widely used in Brain-Computer Interfaces (BCIs). SSVEP-BCIs have advantages of high classification accuracy, high information transfer rate, and strong anti-interference ability. Traditional studies mostly used low/medium frequency SSVEPs as system control signals. However, visual flickers with low/medium frequencies are uncomfortable, and even cause visual fatigue and epilepsy seizure. High-frequency SSVEP is a promising approach to solve these problems, but its miniature amplitude and low signal-to-noise ratio (SNR) would pose great challenges for target recognition. This study developed an innovative BCI paradigm to enhance the SNR of high-frequency SSVEP, which is named Steady-State asymmetrically Visual Evoked Potential (SSaVEP). Ten characters were encoded by ten couples of asymmetric flickers whose durations only lasted one second and frequencies ranged from 31 to 40 Hz with a step of 1 Hz. Discriminative canonical pattern matching (DCPM) was used to decode the high-frequency SSaVEP signals. Four subjects participated in the offline experiment. As a result, the accuracy achieved an average of 87.5% with a peak of 97.1%. The simulated online information transfer rate reached 87.2 bits/min on average and 111.2 bits/min for maximum. The results of this study demonstrate the high-frequency SSaVEP paradigm is a promising approach to alleviate the discomfort caused by visual stimuli and thereby can broaden the applications of BCIs.
Collapse
|
57
|
Han J, Xu M, Wang Y, Tang J, Liu M, An X, Jung TP, Ming D. 'Write' but not 'spell' Chinese characters with a BCI-controlled robot .. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:4741-4744. [PMID: 33019050 DOI: 10.1109/embc44109.2020.9175275] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
Abstract
Visual brain-computer interface (BCI) systems have made tremendous process in recent years. It has been demonstrated to perform well in spelling words. However, different from spelling English words in one-dimension sequences, Chinese characters are often written in a two-dimensional structure. Previous studies had never investigated how to use BCI to 'write' but not 'spell' Chinese characters. This study developed an innovative BCI-controlled robot for writing Chinese characters. The BCI system contained 108 commands displayed in a 9*12 array. A pixel-based writing method was proposed to map the starting point and ending point of each stroke of Chinese characters to the array. Connecting the starting and ending points for each stroke can make up any Chinese character. The large command set was encoded by the hybrid P300 and SSVEP features efficiently, in which each output needed only 1s of EEG data. The task-related component analysis was used to decode the combined features. Five subjects participated in this study and achieved an average accuracy of 87.23% and a maximal accuracy of 100%. The corresponding information transfer rate was 56.85 bits/min and 71.10 bits/min, respectively. The BCI-controlled robotic arm could write a Chinese character '' with 16 strokes within 5.7 seconds for the best subject. The demo video can be found at https://www.youtube.com/watch?v=A1w-e2dBGl0. The study results demonstrated that the proposed BCI-controlled robot is efficient for writing ideogram (e.g. Chinese characters) and phonogram (e.g. English letter), leading to broad prospects for real-world applications of BCIs.
Collapse
|
58
|
Wang L, Xu M, Mei J, Han J, Wang Y, Jung TP, Ming D. Enhancing performance of SSVEP-based BCI by unsupervised learning information from test trials .. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:3359-3362. [PMID: 33018724 DOI: 10.1109/embc44109.2020.9176851] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
Abstract
Steady-State Visual Evoked Potentials (SSVEPs) have become one of the most used neural signals for brain- computer interfaces (BCIs) due to their stability and high signal- to-noise rate. However, the performance of SSVEP-based BCIs would degrade with a few training samples. This study was proposed to enhance the detection of SSVEP by combining the supervised learning information from training samples and the unsupervised learning information from the trial to be tested. A new method, i.e. cyclic shift trials (CST), was proposed to generate new calibration samples from the test data, which were furtherly used to create the templates and spatial filters of task- related component analysis (TRCA). The test-trial templates and spatial filters were combined with training-sample templates and spatial filters to recognize SSVEP. The proposed algorithm was tested on a benchmark dataset. As a result, it reached significantly higher classification accuracy than traditional TRCA when only two training samples were used. Speciflcally, the accuracy was improved by 9.5% for 0.7s data. Therefore, this study demonstrates CST is effective to improve the performance of SSVEP-BCI.
Collapse
|
59
|
Chiang KJ, Nakanishi M, Jung TP. Statistically Optimized Spatial Filtering in Decoding Steady-State Visual Evoked Potentials Based on Task-Related Component Analysis. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2020:3070-3073. [PMID: 33018653 DOI: 10.1109/embc44109.2020.9176205] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
Task-related component analysis (TRCA) has been the most effective spatial filtering method in implementing high-speed brain-computer interfaces (BCIs) based on steady-state visual evoked potentials (SSVEPs). TRCA is a data-driven method, in which spatial filters are optimized to maximize inter-trial covariance of time-locked electroencephalographic (EEG) data, formulated as a generalized eigenvalue problem. Although multiple eigenvectors can be obtained by TRCA, the traditional TRCA-based SSVEP detection considered only one that corresponds to the largest eigenvalue to reduce its computational cost. This study proposes using multiple eigen-vectors to classify SSVEPs. Specifically, this study integrates a task consistency test, which statistically identifies whether the component reconstructed by each eigenvector is task-related or not, with the TRCA-based SSVEP detection method. The proposed method was evaluated by using a 12-class SSVEP dataset recorded from 10 subjects. The study results indicated that the task consistency test usually identified and suggested more than one eigenvectors (i.e., spatial filters). Further, the use of additional spatial filters significantly improved the classification accuracy of the TRCA-based SSVEP detection.
Collapse
|
60
|
Zhang X, Wu D, Ding L, Luo H, Lin CT, Jung TP, Chavarriaga R. Tiny noise, big mistakes: adversarial perturbations induce errors in brain–computer interface spellers. Natl Sci Rev 2020; 8:nwaa233. [PMID: 34691612 PMCID: PMC8288388 DOI: 10.1093/nsr/nwaa233] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/02/2020] [Revised: 07/17/2020] [Accepted: 08/31/2020] [Indexed: 11/25/2022] Open
Abstract
An electroencephalogram (EEG)-based brain–computer interface (BCI) speller allows a user to input text to a computer by thought. It is particularly useful to severely disabled individuals, e.g. amyotrophic lateral sclerosis patients, who have no other effective means of communication with another person or a computer. Most studies so far focused on making EEG-based BCI spellers faster and more reliable; however, few have considered their security. This study, for the first time, shows that P300 and steady-state visual evoked potential BCI spellers are very vulnerable, i.e. they can be severely attacked by adversarial perturbations, which are too tiny to be noticed when added to EEG signals, but can mislead the spellers to spell anything the attacker wants. The consequence could range from merely user frustration to severe misdiagnosis in clinical applications. We hope our research can attract more attention to the security of EEG-based BCI spellers, and more broadly, EEG-based BCIs, which has received little attention before.
Collapse
|
61
|
Jin J, Chen Z, Xu R, Miao Y, Wang X, Jung TP. Developing a Novel Tactile P300 Brain-Computer Interface With a Cheeks-Stim Paradigm. IEEE Trans Biomed Eng 2020; 67:2585-2593. [DOI: 10.1109/tbme.2020.2965178] [Citation(s) in RCA: 55] [Impact Index Per Article: 13.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
62
|
Liang CP, She HC, Huang LY, Chou WC, Chen SC, Jung TP. Human Brain Dynamics Reflect the Correctness and Presentation Modality of Physics Concept Memory Retrieval. Front Hum Neurosci 2020; 14:331. [PMID: 33110406 PMCID: PMC7488981 DOI: 10.3389/fnhum.2020.00331] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/21/2020] [Accepted: 07/27/2020] [Indexed: 11/13/2022] Open
Abstract
Human memory retrieval is the core cognitive process of the human brain whenever it is processing the information. Less study has focused on exploring the neural correlates of the memory retrieval of scientific concepts when presented in word and picture modalities. Fewer studies have investigated the differences in the involved brain regions and how the brain dynamics in these regions would associate with the accuracy of the memory retrieval process. Therefore, this study specifically focused on investigating the human brain dynamics of participants when they retrieve physics concepts in word vs. pictorial modalities, and whether electroencephalogram (EEG) activities can predict the correctness of the retrieval of physics concepts. The results indicated that word modality induced a significant stronger right frontal theta augmentation than pictorial modality during the physics concepts retrieval process, whereas the picture modality induced a significantly greater right parietal alpha suppression than the word modality throughout the retrieval process spurred by the physics concept presentations. In addition, greater frontal midline theta augmentation was observed for incorrect responses than the correct responses during retrieve physics concepts. Moreover, the frontal midline theta power has greater negative predictive power for predicting the accuracy of physics concepts retrieval. In summary, the participants were more likely to retrieve physics concepts correctly if a lower amount of theta were allocated during the maintaining period from 2,000 ms through 3,500 ms before making responses. It provides insight for our future application of brain computer interface (BCI) in real-time science learning. This study implies that the lower frontal midline theta power is associated with a lower degree of cognitive control and active maintenance of representations as participants approach a correct answer.
Collapse
|
63
|
Wen D, Yuan J, Zhou Y, Xu J, Song H, Liu Y, Xu Y, Jung TP. The EEG Signal Analysis for Spatial Cognitive Ability Evaluation Based on Multivariate Permutation Conditional Mutual Information-Multi-Spectral Image. IEEE Trans Neural Syst Rehabil Eng 2020; 28:2113-2122. [PMID: 32833638 DOI: 10.1109/tnsre.2020.3018959] [Citation(s) in RCA: 6] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
This study aims to find an effective method to evaluate the efficacy of cognitive training of spatial memory under a virtual reality environment, by classifying the EEG signals of subjects in the early and late stages of spatial cognitive training. This study proposes a new EEG signal analysis method based on Multivariate Permutation Conditional Mutual Information-Multi-Spectral Image (MPCMIMSI). This method mainly considers the relationship between the coupled features of EEG signals in different channel pairs and transforms the multivariate permutation conditional mutual information features into multi-spectral images. Then, a convolutional neural networks (CNN) model classifies the resultant image data into different stages of cognitive training to objectively assess the efficacy of the training. Compared to the multi-spectral image transformation method based on Granger causality analysis (GCA) and permutation conditional mutual information (PCMI), the MPCMIMSI led to better classification performance, which can be as high as 95% accuracy. More specifically, the Theta-Beta2-Gamma-band combination has the best accuracy. The proposed MPCMIMSI method outperforms the multi-spectral image transformation methods based on GCA and PCMI in terms of classification performance. The MPCMIMSI feature in the Theta-Beta2-Gamma band is an effective biomarker for assessing the efficacy of spatial memory training. The proposed EEG feature-extraction method based on MPCMIMSI offers a new window to characterize spatial information of the noninvasive EEG recordings and might apply to assessing other brain functions.
Collapse
|
64
|
Wong CM, Wang Z, Wang B, Lao KF, Rosa A, Xu P, Jung TP, Chen CLP, Wan F. Inter- and Intra-Subject Transfer Reduces Calibration Effort for High-Speed SSVEP-Based BCIs. IEEE Trans Neural Syst Rehabil Eng 2020; 28:2123-2135. [PMID: 32841119 DOI: 10.1109/tnsre.2020.3019276] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
OBJECTIVE Steady-state visual evoked potential (SSVEP)-based brain-computer interfaces (BCIs) that can deliver a high information transfer rate (ITR) usually require subject's calibration data to learn the class- and subject-specific model parameters (e.g. the spatial filters and SSVEP templates). Normally, the amount of the calibration data for learning is proportional to the number of classes (or visual stimuli), which could be huge and consequently lead to a time-consuming calibration. This study presents a transfer learning scheme to substantially reduce the calibration effort. METHODS Inspired by the parameter-based and instance-based transfer learning techniques, we propose a subject transfer based canonical correlation analysis (stCCA) method which utilizes the knowledge within subject and between subjects, thus requiring few calibration data from a new subject. RESULTS The evaluation study on two SSVEP datasets (from Tsinghua and UCSD) shows that the stCCA method performs well with only a small amount of calibration data, providing an ITR at 198.18±59.12 (bits/min) with 9 calibration trials in the Tsinghua dataset and 111.04±57.24 (bits/min) with 3 trials in the UCSD dataset. Such performances are comparable to those from using the multi-stimulus CCA (msCCA) and the ensemble task-related component analysis (eTRCA) methods with the minimally required calibration data (i.e., at least 40 trials in the Tsinghua dataset and at least 12 trials in the UCSD dataset), respectively. CONCLUSION Inter- and intra-subject transfer helps the recognition method achieve high ITR with extremely little calibration effort. SIGNIFICANCE The proposed approach saves much calibration effort without sacrificing the ITR, which would be significant for practical SSVEP-based BCIs.
Collapse
|
65
|
Zhang HY, Stevenson CE, Jung TP, Ko LW. Stress-Induced Effects in Resting EEG Spectra Predict the Performance of SSVEP-Based BCI. IEEE Trans Neural Syst Rehabil Eng 2020; 28:1771-1780. [PMID: 32746309 DOI: 10.1109/tnsre.2020.3005771] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Most research in Brain-Computer-Interfaces (BCI) focuses on technologies to improve accuracy and speed. Little has been done on the effects of subject variability, both across individuals and within the same individual, on BCI performance. For example, stress, arousal, motivation, and fatigue can all affect the electroencephalogram (EEG) signals used by a BCI, which in turn impacts performance. Overcoming the impact of such user variability on BCI performance is an impending and inevitable challenge for routine applications of BCIs in the real world. To systematically explore the factors affecting BCI performance, this study embeds a Steady-State Visually Evoked Potential (SSVEP) based BCI into a "game with a purpose" (GWAP) to obtain data over significant lengths of time, under both high- and low-stress conditions. Ten healthy volunteers played a GWAP that resembles popular match-three games, such as Jewel Quest, Zoo Boom, or Candy Crush. We recorded the target search time, target search accuracy, and EEG signals during gameplay to investigate the impacts of stress on EEG signals and BCI performance. We used Canonical Correlation Analysis (CCA) to determine whether the subject had found and attended to the correct target. The experimental results show that SSVEP target-classification accuracy is reduced by stress. We also found a negative correlation between EEG spectra and the SNR of EEG in the frontal and occipital regions during gameplay, with a larger negative correlation for the high-stress conditions. Furthermore, CCA also showed that when the EEG alpha and theta power increased, the search accuracy decreased, and the spectral amplitude drop was more evident under the high-stress situation. These results provide new, valuable insights into research on how to improve the robustness of BCIs in real-world applications.
Collapse
|
66
|
Wang H, Liu X, Hu H, Wan F, Li T, Gao L, Bezerianos A, Sun Y, Jung TP. Dynamic Reorganization of Functional Connectivity Unmasks Fatigue Related Performance Declines in Simulated Driving. IEEE Trans Neural Syst Rehabil Eng 2020; 28:1790-1799. [DOI: 10.1109/tnsre.2020.2999599] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/05/2022]
|
67
|
Xiao X, Xu M, Jin J, Wang Y, Jung TP, Ming D. Discriminative Canonical Pattern Matching for Single-Trial Classification of ERP Components. IEEE Trans Biomed Eng 2020; 67:2266-2275. [DOI: 10.1109/tbme.2019.2958641] [Citation(s) in RCA: 51] [Impact Index Per Article: 12.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
|
68
|
Panwar S, Rad P, Jung TP, Huang Y. Modeling EEG Data Distribution With a Wasserstein Generative Adversarial Network to Predict RSVP Events. IEEE Trans Neural Syst Rehabil Eng 2020; 28:1720-1730. [PMID: 32746311 DOI: 10.1109/tnsre.2020.3006180] [Citation(s) in RCA: 13] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Electroencephalography (EEG) data are difficult to obtain due to complex experimental setups and reduced comfort with prolonged wearing. This poses challenges to train powerful deep learning model with the limited EEG data. Being able to generate EEG data computationally could address this limitation. We propose a novel Wasserstein Generative Adversarial Network with gradient penalty (WGAN-GP) to synthesize EEG data. This network addresses several modeling challenges of simulating time-series EEG data including frequency artifacts and training instability. We further extended this network to a class-conditioned variant that also includes a classification branch to perform event-related classification. We trained the proposed networks to generate one and 64-channel data resembling EEG signals routinely seen in a rapid serial visual presentation (RSVP) experiment and demonstrated the validity of the generated samples. We also tested intra-subject cross-session classification performance for classifying the RSVP target events and showed that class-conditioned WGAN-GP can achieve improved event-classification performance over EEGNet.
Collapse
|
69
|
Ko LW, Komarov O, Lai WK, Liang WG, Jung TP. Eyeblink recognition improves fatigue prediction from single-channel forehead EEG in a realistic sustained attention task. J Neural Eng 2020; 17:036015. [DOI: 10.1088/1741-2552/ab909f] [Citation(s) in RCA: 20] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/11/2022]
|
70
|
Xu M, Meng J, Yu H, Jung TP, Ming D. Dynamic Brain Responses Modulated by Precise Timing Prediction in an Opposing Process. Neurosci Bull 2020; 37:70-80. [PMID: 32548801 DOI: 10.1007/s12264-020-00527-1] [Citation(s) in RCA: 3] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/09/2019] [Accepted: 02/11/2020] [Indexed: 01/04/2023] Open
Abstract
The brain function of prediction is fundamental for human beings to shape perceptions efficiently and successively. Through decades of effort, a valuable brain activation map has been obtained for prediction. However, much less is known about how the brain manages the prediction process over time using traditional neuropsychological paradigms. Here, we implemented an innovative paradigm for timing prediction to precisely study the temporal dynamics of neural oscillations. In the experiment recruiting 45 participants, expectation suppression was found for the overall electroencephalographic activity, consistent with previous hemodynamic studies. Notably, we found that N1 was positively associated with predictability while N2 showed a reversed relation to predictability. Furthermore, the matching prediction had a similar profile with no timing prediction, both showing an almost saturated N1 and an absence of N2. The results indicate that the N1 process showed a 'sharpening' effect for predictable inputs, while the N2 process showed a 'dampening' effect. Therefore, these two paradoxical neural effects of prediction, which have provoked wide confusion in accounting for expectation suppression, actually co-exist in the procedure of timing prediction but work in separate time windows. These findings strongly support a recently-proposed opposing process theory.
Collapse
|
71
|
Xu M, Zhou X, Xiao X, Wang Y, Jung TP, Ming D. Effects of stimulus position on the classification of miniature asymmetric VEPs for brain-computer interfaces. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2019:5956-5959. [PMID: 31947204 DOI: 10.1109/embc.2019.8857789] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
The speed of visual brain-computer interfaces (BCIs) has been greatly improved in recent years. However, traditional visual BCI paradigm requires users to directly gaze at the intensive flickering items, which would cause severe problems in practical applications, such as visual fatigue and excessive visual resources consumption. A promising solution is to use small visual stimuli outside the central visual area to encode instructions, which had been demonstrated to be effective in our previous study. This study aims to further investigate the effects of stimulus position on the classification of miniature asymmetric visual evoked potentials (aVEPs). Small peripheral visual stimuli were designed with different eccentricities (1° and 2°) and directions (0°, 45°, 90°, 135°, 180°, -135°, -90°, and -45°) to induce different kinds of miniature aVEPs. Five subjects participated in this experiment. Discriminative canonical pattern matching (DCPM) was used to classify all possible pairs of miniature aVEPs. Study results showed that visual stimuli with less eccentricity could induce more distinct miniature aVEPs. The highest single-trial accuracy achieved was about 83% for the binary classifications of miniature aVEPs pairs corresponding to (1°, -135°) Vs (1°, 0°), (1°, -45°) Vs (1°, -135°) and (1°, -45°) Vs (1°, 180°). This finding is very important for the design and development of the miniature aVEPs-based BCIs.
Collapse
|
72
|
Nakanishi M, Xu M, Wang Y, Chiang KJ, Han J, Jung TP. Questionable Classification Accuracy Reported in "Designing a Sum of Squared Correlations Framework for Enhancing SSVEP-Based BCIs". IEEE Trans Neural Syst Rehabil Eng 2020; 28:1042-1043. [PMID: 32078554 DOI: 10.1109/tnsre.2020.2974272] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/09/2022]
Abstract
This commentary presents a replication study to verify the effectiveness of a sum of squared correlations (SSCOR)-based steady-state visual evoked potentials (SSVEPs) decoding method proposed by Kumar et al.. We implemented the SSCOR-based method in accordance with their descriptions and estimated its classification accuracy using a benchmark SSVEP dataset with cross validation. Our results showed significantly lower classification accuracy compared with the ones reported in Kumar et al.'s study. We further investigated the sources of performance discrepancy by simulating data leakage between training and test datasets. The classification performance of the simulation was remarkably similar to those reported by Kumar et al.. We, therefore, question the validity of evaluation and conclusions drawn in Kumar et al.'s study.
Collapse
|
73
|
Komarov O, Ko LW, Jung TP. Associations Among Emotional State, Sleep Quality, and Resting-State EEG Spectra: A Longitudinal Study in Graduate Students. IEEE Trans Neural Syst Rehabil Eng 2020; 28:795-804. [PMID: 32070988 DOI: 10.1109/tnsre.2020.2972812] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
University students are routinely influenced by a variety of natural stressors and experience irregular sleep-wake cycles caused by the necessity to trade sleep for studying while dealing with academic assignments. Often these factors result in long-term issues with daytime sleepiness, emotional instability, and mental exhaustion, which may lead to difficulties in the educational process. This study introduces the Daily Sampling System (DSS) implemented as a smartphone application, which combines a set of self-assessment scales for evaluating variations in the emotional state and sleep quality throughout a full academic term. In addition to submitting the daily sampling scores, the participants regularly filled in the Depression, Anxiety, and Stress Scales (DASS) reports and took part in resting-state EEG data recording immediately after report completion. In total, this study collected 1835 daily samples and 94 combined DASS with EEG datasets from 18 university students (aged 23-27 years), with 79.3± 15.3% response ratio in submitting the daily reports during an academic semester. The results of pairwise testing and multiple regression analysis demonstrate that the daily level of self-perceived fatigue correlates positively with stress, daytime sleepiness, and negatively with alertness on awakening, self-evaluated sleep quality, and sleep duration. The spectral analysis of the EEG data reveals a significant increase in the resting-state spectral power density across the theta and low-alpha frequency bands associated with increased levels of anxiety and stress. Additionally, the state of depression was accompanied by an intensification of high-frequency EEG activity over the temporal regions. No significant differences in prefrontal alpha power asymmetry were observed under the described experimental conditions while comparing the states of calmness and emotional arousal of the participants for the three conditions of depression, anxiety, and stress.
Collapse
|
74
|
Xiao X, Xu M, Wang Y, Jung TP, Ming D. A comparison of classification methods for recognizing single-trial P300 in brain-computer interfaces .. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2020; 2019:3032-3035. [PMID: 31946527 DOI: 10.1109/embc.2019.8857521] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
P300s are one of the most popular and robust control signals for brain-computer interfaces (BCIs). Fast classifying P300s is vital for the good performance of P300-based BCIs. However, due to noisy background electroencephalography (EEG) environments, current P300-based BCI systems need to collect multiple trials for a reliable output, which is inefficient. This study compared a recently developed algorithm, i.e. discriminative canonical pattern matching (DCPM), with five traditional classification methods,i.e. linear discriminant analysis (LDA), stepwise LDA, Bayesian LDA, shrinkage LDA and spatial-temporal discriminant analysis (STDA), for the detection of single-trial P300s. Eight subjects participated in the classical P300-speller experiments. Study results showed that the DCPM significantly outperformed the other traditional methods in single-trial P300 classification even with small training samples, suggesting the DCPM is a promising classification algorithm for the P300-based BCI.
Collapse
|
75
|
Siddharth S, Jung TP, Sejnowski TJ. Impact of Affective Multimedia Content on the Electroencephalogram and Facial Expressions. Sci Rep 2019; 9:16295. [PMID: 31705031 PMCID: PMC6841664 DOI: 10.1038/s41598-019-52891-2] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/28/2019] [Accepted: 10/24/2019] [Indexed: 11/24/2022] Open
Abstract
Most of the research in the field of affective computing has focused on detecting and classifying human emotions through electroencephalogram (EEG) or facial expressions. Designing multimedia content to evoke certain emotions has been largely motivated by manual rating provided by users. Here we present insights from the correlation of affective features between three modalities namely, affective multimedia content, EEG, and facial expressions. Interestingly, low-level Audio-visual features such as contrast and homogeneity of the video and tone of the audio in the movie clips are most correlated with changes in facial expressions and EEG. We also detect the regions associated with the human face and the brain (in addition to the EEG frequency bands) that are most representative of affective responses. The computational modeling between the three modalities showed a high correlation between features from these regions and user-reported affective labels. Finally, the correlation between different layers of convolutional neural networks with EEG and Face images as input provides insights into human affection. Together, these findings will assist in (1) designing more effective multimedia contents to engage or influence the viewers, (2) understanding the brain/body bio-markers of affection, and (3) developing newer brain-computer interfaces as well as facial-expression-based algorithms to read emotional responses of the viewers.
Collapse
|