1
|
Lei D, Dong C, Guo H, Ma P, Liu H, Bao N, Kang H, Chen X, Wu Y. A fused multi-subfrequency bands and CBAM SSVEP-BCI classification method based on convolutional neural network. Sci Rep 2024; 14:8616. [PMID: 38616204 PMCID: PMC11016546 DOI: 10.1038/s41598-024-59348-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/09/2024] [Accepted: 04/09/2024] [Indexed: 04/16/2024] Open
Abstract
For the brain-computer interface (BCI) system based on steady-state visual evoked potential (SSVEP), it is difficult to obtain satisfactory classification performance for short-time window SSVEP signals by traditional methods. In this paper, a fused multi-subfrequency bands and convolutional block attention module (CBAM) classification method based on convolutional neural network (CBAM-CNN) is proposed for discerning SSVEP-BCI tasks. This method extracts multi-subfrequency bands SSVEP signals as the initial input of the network model, and then carries out feature fusion on all feature inputs. In addition, CBAM is embedded in both parts of the initial input and feature fusion for adaptive feature refinement. To verify the effectiveness of the proposed method, this study uses the datasets of Inner Mongolia University of Technology (IMUT) and Tsinghua University (THU) to evaluate the performance of the proposed method. The experimental results show that the highest accuracy of CBAM-CNN reaches 0.9813 percentage point (pp). Within 0.1-2 s time window, the accuracy of CBAM-CNN is 0.0201-0.5388 (pp) higher than that of CNN, CCA-CWT-SVM, CCA-SVM, CCA-GNB, FBCCA, and CCA. Especially in the short-time window range of 0.1-1 s, the performance advantage of CBAM-CNN is more significant. The maximum information transmission rate (ITR) of CBAM-CNN is 503.87 bit/min, which is 227.53 bit/min-503.41 bit/min higher than the above six EEG decoding methods. The study further results show that CBAM-CNN has potential application value in SSVEP decoding.
Collapse
Affiliation(s)
- Dongyang Lei
- College of Electric Power, Inner Mongolia University of Technology, Hohhot, 010080, China
- Intelligent Energy Technology and Equipment Engineering Research Centre of Colleges and Universities in Inner Mongolia Autonomous Region, Hohhot, 010051, China
| | - Chaoyi Dong
- College of Electric Power, Inner Mongolia University of Technology, Hohhot, 010080, China.
- Intelligent Energy Technology and Equipment Engineering Research Centre of Colleges and Universities in Inner Mongolia Autonomous Region, Hohhot, 010051, China.
- Engineering Research Center of Large Energy Storage Technology, Ministry of Education, Hohhot, 010080, China.
- Inner Mongolia Academy of Science and Technology, Hohhot, 010010, China.
| | - Hongfei Guo
- Inner Mongolia Academy of Science and Technology, Hohhot, 010010, China.
| | - Pengfei Ma
- College of Electric Power, Inner Mongolia University of Technology, Hohhot, 010080, China
- Intelligent Energy Technology and Equipment Engineering Research Centre of Colleges and Universities in Inner Mongolia Autonomous Region, Hohhot, 010051, China
| | - Huanzi Liu
- College of Electric Power, Inner Mongolia University of Technology, Hohhot, 010080, China
- Intelligent Energy Technology and Equipment Engineering Research Centre of Colleges and Universities in Inner Mongolia Autonomous Region, Hohhot, 010051, China
| | - Naqin Bao
- College of Electric Power, Inner Mongolia University of Technology, Hohhot, 010080, China
- Intelligent Energy Technology and Equipment Engineering Research Centre of Colleges and Universities in Inner Mongolia Autonomous Region, Hohhot, 010051, China
| | - Hongzhuo Kang
- College of Electric Power, Inner Mongolia University of Technology, Hohhot, 010080, China
- Intelligent Energy Technology and Equipment Engineering Research Centre of Colleges and Universities in Inner Mongolia Autonomous Region, Hohhot, 010051, China
| | - Xiaoyan Chen
- College of Electric Power, Inner Mongolia University of Technology, Hohhot, 010080, China
- Intelligent Energy Technology and Equipment Engineering Research Centre of Colleges and Universities in Inner Mongolia Autonomous Region, Hohhot, 010051, China
- Engineering Research Center of Large Energy Storage Technology, Ministry of Education, Hohhot, 010080, China
- Inner Mongolia Academy of Science and Technology, Hohhot, 010010, China
| | - Yi Wu
- Inner Mongolia Academy of Science and Technology, Hohhot, 010010, China
| |
Collapse
|
2
|
Wu X, Li G, Jiang S, Wellington S, Liu S, Wu Z, Metcalfe B, Chen L, Zhang D. Decoding Continuous Kinetic Information of Grasp from Stereo-electroencephalographic (SEEG) Recordings. J Neural Eng 2022; 19. [PMID: 35395645 DOI: 10.1088/1741-2552/ac65b1] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/10/2021] [Accepted: 04/08/2022] [Indexed: 11/11/2022]
Abstract
OBJECTIVE Brain-computer interfaces (BCIs) have the potential to bypass damaged neural pathways and restore functionality lost due to injury or disease. Approaches to decoding kinematic information are well documented; however, the decoding of kinetic information has received less attention. Additionally, the possibility of using stereo-electroencephalography (SEEG) for kinetic decoding during hand grasping tasks is still largely unknown. Thus, the objective of this paper is to demonstrate kinetic parameter decoding using SEEG in patients performing a grasping task with two different force levels under two different ascending rates. APPROACH Temporal-spectral representations were studied to investigate frequency modulation under different force tasks. Then, force amplitude was decoded from SEEG recordings using multiple decoders, including a linear model, a partial least squares (PLS) model, an unscented Kalman filter, and three deep learning models (shallow convolutional neural network, deep convolutional neural network and the proposed CNN+RNN neural network). MAIN RESULTS The current study showed that: 1) for some channel, both low-frequency modulation (event-related desynchronization (ERD)) and high-frequency modulation (event-related synchronization (ERS)) were sustained during prolonged force holding periods; 2) continuously changing grasp force can be decoded from the SEEG signals; 3) the novel CNN+RNN deep learning model achieved the best decoding performance, with the predicted force magnitude closely aligned to the ground truth under different force amplitudes and changing rates. SIGNIFICANCE This work verified the possibility of decoding continuously changing grasp force using SEEG recordings. The result presented in this study demonstrated the potential of SEEG recordings for future BCI application.
Collapse
Affiliation(s)
- Xiaolong Wu
- Electric, Electronic and Engineering, University of Bath, Pulteney Court PD42.2,Pulteney Road,BA2 4HL, Bath, Bath, Somerset, BA2 4HL, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Guangye Li
- Shanghai Jiao Tong University, Shanghai JiaoTong university, Shanghai, China, Shanghai, 200240, CHINA
| | - Shize Jiang
- Fudan University Huashan Hospital Department of Neurosurgery, Fudan University Huanshan hospital, Shanghai, Shanghai, 201906, CHINA
| | - Scott Wellington
- University of Bath, University of Bath, Bath, UK, Bath, Bath and North East Somer, BA2 7AY, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Shengjie Liu
- Shanghai Jiao Tong University, Shanghai jIaotong University, Shanghai, China, Shanghai, 200240, CHINA
| | - Zehan Wu
- Huashan Hospital Fudan University, Huashan hospital, Shanghai, China, Shanghai, Shanghai, 200040, CHINA
| | - Benjamin Metcalfe
- University of Bath, University of Bath, UK, Bath, Bath and North East Somer, BA2 7AY, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| | - Liang Chen
- Fudan University Huashan Hospital Department of Neurosurgery, Fudan University Huashan Hospital, Shanghai, China, Shanghai, Shanghai, 201906, CHINA
| | - Dingguo Zhang
- University of Bath, University of Bath, UK, Bath, Somerset, BA2 4HL, UNITED KINGDOM OF GREAT BRITAIN AND NORTHERN IRELAND
| |
Collapse
|
3
|
Bouton C, Bhagat N, Chandrasekaran S, Herrero J, Markowitz N, Espinal E, Kim JW, Ramdeo R, Xu J, Glasser MF, Bickel S, Mehta A. Decoding Neural Activity in Sulcal and White Matter Areas of the Brain to Accurately Predict Individual Finger Movement and Tactile Stimuli of the Human Hand. Front Neurosci 2021; 15:699631. [PMID: 34483823 PMCID: PMC8415782 DOI: 10.3389/fnins.2021.699631] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/23/2021] [Accepted: 07/22/2021] [Indexed: 01/01/2023] Open
Abstract
Millions of people worldwide suffer motor or sensory impairment due to stroke, spinal cord injury, multiple sclerosis, traumatic brain injury, diabetes, and motor neuron diseases such as ALS (amyotrophic lateral sclerosis). A brain-computer interface (BCI), which links the brain directly to a computer, offers a new way to study the brain and potentially restore impairments in patients living with these debilitating conditions. One of the challenges currently facing BCI technology, however, is to minimize surgical risk while maintaining efficacy. Minimally invasive techniques, such as stereoelectroencephalography (SEEG) have become more widely used in clinical applications in epilepsy patients since they can lead to fewer complications. SEEG depth electrodes also give access to sulcal and white matter areas of the brain but have not been widely studied in brain-computer interfaces. Here we show the first demonstration of decoding sulcal and subcortical activity related to both movement and tactile sensation in the human hand. Furthermore, we have compared decoding performance in SEEG-based depth recordings versus those obtained with electrocorticography electrodes (ECoG) placed on gyri. Initial poor decoding performance and the observation that most neural modulation patterns varied in amplitude trial-to-trial and were transient (significantly shorter than the sustained finger movements studied), led to the development of a feature selection method based on a repeatability metric using temporal correlation. An algorithm based on temporal correlation was developed to isolate features that consistently repeated (required for accurate decoding) and possessed information content related to movement or touch-related stimuli. We subsequently used these features, along with deep learning methods, to automatically classify various motor and sensory events for individual fingers with high accuracy. Repeating features were found in sulcal, gyral, and white matter areas and were predominantly phasic or phasic-tonic across a wide frequency range for both HD (high density) ECoG and SEEG recordings. These findings motivated the use of long short-term memory (LSTM) recurrent neural networks (RNNs) which are well-suited to handling transient input features. Combining temporal correlation-based feature selection with LSTM yielded decoding accuracies of up to 92.04 ± 1.51% for hand movements, up to 91.69 ± 0.49% for individual finger movements, and up to 83.49 ± 0.72% for focal tactile stimuli to individual finger pads while using a relatively small number of SEEG electrodes. These findings may lead to a new class of minimally invasive brain-computer interface systems in the future, increasing its applicability to a wide variety of conditions.
Collapse
Affiliation(s)
- Chad Bouton
- Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States.,Hofstra-Northwell Medical School, New York, NY, United States
| | - Nikunj Bhagat
- Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States
| | - Santosh Chandrasekaran
- Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States
| | - Jose Herrero
- Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States.,Department of Neurosurgery, Northwell Health, New York, NY, United States
| | - Noah Markowitz
- Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States
| | - Elizabeth Espinal
- Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States
| | - Joo-Won Kim
- Department of Radiology and Psychiatry, Baylor College of Medicine, Houston, TX, United States
| | - Richard Ramdeo
- Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States
| | - Junqian Xu
- Department of Radiology and Psychiatry, Baylor College of Medicine, Houston, TX, United States
| | - Matthew F Glasser
- Department of Radiology and Neuroscience, Washington University in St. Louis, St. Louis, MO, United States
| | - Stephan Bickel
- Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States.,Hofstra-Northwell Medical School, New York, NY, United States.,Department of Neurosurgery, Northwell Health, New York, NY, United States.,Department of Neurology, Northwell Health, New York, NY, United States
| | - Ashesh Mehta
- Feinstein Institutes for Medical Research at Northwell Health, New York, NY, United States.,Institute of Bioelectronic Medicine, Feinstein Institutes for Medical Research, New York, NY, United States.,Hofstra-Northwell Medical School, New York, NY, United States.,Department of Neurosurgery, Northwell Health, New York, NY, United States
| |
Collapse
|