1
|
Wu H, Li F, Chu W, Li Y, Niu Y, Shi G, Zhang L, Chen Y. Semantic image sorting method for RSVP presentation. J Neural Eng 2024; 21:036018. [PMID: 38688262 DOI: 10.1088/1741-2552/ad4593] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/21/2023] [Accepted: 04/30/2024] [Indexed: 05/02/2024]
Abstract
Objective.The rapid serial visual presentation (RSVP) paradigm, which is based on the electroencephalogram (EEG) technology, is an effective approach for object detection. It aims to detect the event-related potentials (ERP) components evoked by target images for rapid identification. However, the object detection performance within this paradigm is affected by the visual disparity between adjacent images in a sequence. Currently, there is no objective metric to quantify this visual difference. Consequently, a reliable image sorting method is required to ensure the generation of a smooth sequence for effective presentation.Approach. In this paper, we propose a novel semantic image sorting method for sorting RSVP sequences, which aims at generating sequences that are perceptually smoother in terms of the human visual experience.Main results. We conducted a comparative analysis between our method and two existing methods for generating RSVP sequences using both qualitative and quantitative assessments. A qualitative evaluation revealed that the sequences generated by our method were smoother in subjective vision and were more effective in evoking stronger ERP components than those generated by the other two methods. Quantitatively, our method generated semantically smoother sequences than the other two methods. Furthermore, we employed four advanced approaches to classify single-trial EEG signals evoked by each of the three methods. The classification results of the EEG signals evoked by our method were superior to those of the other two methods.Significance. In summary, the results indicate that the proposed method can significantly enhance the object detection performance in RSVP-based sequences.
Collapse
Affiliation(s)
- Hao Wu
- Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial Intelligence, Xidian University, Xi'an, People's Republic of China
| | - Fu Li
- Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial Intelligence, Xidian University, Xi'an, People's Republic of China
| | - Wenlong Chu
- Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial Intelligence, Xidian University, Xi'an, People's Republic of China
| | - Yang Li
- Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial Intelligence, Xidian University, Xi'an, People's Republic of China
| | - Yi Niu
- Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial Intelligence, Xidian University, Xi'an, People's Republic of China
| | - Guangming Shi
- Key Laboratory of Intelligent Perception and Image Understanding of Ministry of Education, School of Artificial Intelligence, Xidian University, Xi'an, People's Republic of China
| | - Lijian Zhang
- Beijing Institute of Mechanical Equipment, Beijing, People's Republic of China
| | - Yuanfang Chen
- Beijing Institute of Mechanical Equipment, Beijing, People's Republic of China
| |
Collapse
|
2
|
Wang X, Li B, Lin Y, Gao X. Multi-source domain adaptation based tempo-spatial convolution network for cross-subject EEG classification in RSVP task. J Neural Eng 2024; 21:016025. [PMID: 38324909 DOI: 10.1088/1741-2552/ad2710] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/20/2023] [Accepted: 02/07/2024] [Indexed: 02/09/2024]
Abstract
Objective.Many subject-dependent methods were proposed for electroencephalogram (EEG) classification in rapid serial visual presentation (RSVP) task, which required a large amount of data from new subject and were time-consuming to calibrate system. Cross-subject classification can realize calibration reduction or zero calibration. However, cross-subject classification in RSVP task is still a challenge.Approach.This study proposed a multi-source domain adaptation based tempo-spatial convolution (MDA-TSC) network for cross-subject RSVP classification. The proposed network consisted of three modules. First, the common feature extraction with multi-scale tempo-spatial convolution was constructed to extract domain-invariant features across all subjects, which could improve generalization of the network. Second, the multi-branch domain-specific feature extraction and alignment was conducted to extract and align domain-specific feature distributions of source and target domains in pairs, which could consider feature distribution differences among source domains. Third, the domain-specific classifier was exploited to optimize the network through loss functions and obtain prediction for the target domain.Main results.The proposed network was evaluated on the benchmark RSVP dataset, and the cross-subject classification results showed that the proposed MDA-TSC network outperformed the reference methods. Moreover, the effectiveness of the MDA-TSC network was verified through both ablation studies and visualization.Significance.The proposed network could effectively improve cross-subject classification performance in RSVP task, and was helpful to reduce system calibration time.
Collapse
Affiliation(s)
- Xuepu Wang
- School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing 100081, People's Republic of China
| | - Bowen Li
- School of Medicine, Tsinghua University, Beijing 100084, People's Republic of China
| | - Yanfei Lin
- School of Integrated Circuits and Electronics, Beijing Institute of Technology, Beijing 100081, People's Republic of China
| | - Xiaorong Gao
- School of Medicine, Tsinghua University, Beijing 100084, People's Republic of China
| |
Collapse
|
3
|
Luo J, Cui W, Xu S, Wang L, Chen H, Li Y. A Cross-Scale Transformer and Triple-View Attention Based Domain-Rectified Transfer Learning for EEG Classification in RSVP Tasks. IEEE Trans Neural Syst Rehabil Eng 2024; 32:672-683. [PMID: 38285586 DOI: 10.1109/tnsre.2024.3359191] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/31/2024]
Abstract
Rapid serial visual presentation (RSVP)-based brain-computer interface (BCI) is a promising target detection technique by using electroencephalogram (EEG) signals. However, existing deep learning approaches seldom considered dependencies of multi-scale temporal features and discriminative multi-view spectral features simultaneously, which limits the representation learning ability of the model and undermine the EEG classification performance. In addition, recent transfer learning-based methods generally failed to obtain transferable cross-subject invariant representations and commonly ignore the individual-specific information, leading to the poor cross-subject transfer performance. In response to these limitations, we propose a cross-scale Transformer and triple-view attention based domain-rectified transfer learning (CST-TVA-DRTL) for the RSVP classification. Specially, we first develop a cross-scale Transformer (CST) to extract multi-scale temporal features and exploit the dependencies of different scales features. Then, a triple-view attention (TVA) is designed to capture spectral features from triple views of multi-channel time-frequency images. Finally, a domain-rectified transfer learning (DRTL) framework is proposed to simultaneously obtain transferable domain-invariant representations and untransferable domain-specific representations, then utilize domain-specific information to rectify domain-invariant representations to adapt to target data. Experimental results on two public RSVP datasets suggests that our CST-TVA-DRTL outperforms the state-of-the-art methods in the RSVP classification task. The source code of our model is publicly available in https://github.com/ljbuaa/CST_TVA_DRTL.
Collapse
|
4
|
Lin S, Jiang J, Huang K, Li L, He X, Du P, Wu Y, Liu J, Li X, Huang Z, Zhou Z, Yu Y, Gao J, Lei M, Wu H. Advanced Electrode Technologies for Noninvasive Brain-Computer Interfaces. ACS NANO 2023; 17:24487-24513. [PMID: 38064282 DOI: 10.1021/acsnano.3c06781] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/27/2023]
Abstract
Brain-computer interfaces (BCIs) have garnered significant attention in recent years due to their potential applications in medical, assistive, and communication technologies. Building on this, noninvasive BCIs stand out as they provide a safe and user-friendly method for interacting with the human brain. In this work, we provide a comprehensive overview of the latest developments and advancements in material, design, and application of noninvasive BCIs electrode technology. We also explore the challenges and limitations currently faced by noninvasive BCI electrode technology and sketch out the technological roadmap from three dimensions: Materials and Design; Performances; Mode and Function. We aim to unite research efforts within the field of noninvasive BCI electrode technology, focusing on the consolidation of shared goals and fostering integrated development strategies among a diverse array of multidisciplinary researchers.
Collapse
Affiliation(s)
- Sen Lin
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Jingjing Jiang
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Kai Huang
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
- State Key Laboratory of Information Photonics and Optical Communications and School of Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
| | - Lei Li
- National Engineering Research Center of Electric Vehicles, Beijing Institute of Technology, Beijing 100081, China
| | - Xian He
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| | - Peng Du
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| | - Yufeng Wu
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| | - Junchen Liu
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
- State Key Laboratory of Information Photonics and Optical Communications and School of Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
| | - Xilin Li
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
- Advanced Institute for Brain and Intelligence, Guangxi University, Nanning 530004, China
| | - Zhibao Huang
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Zenan Zhou
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Yuanhang Yu
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Jiaxin Gao
- School of Physical Science and Technology, Guangxi University, Nanning 530004, China
| | - Ming Lei
- State Key Laboratory of Information Photonics and Optical Communications and School of Science, Beijing University of Posts and Telecommunications, Beijing 100876, China
| | - Hui Wu
- State Key Laboratory of New Ceramics and Fine Processing, School of Materials Science and Engineering, Tsinghua University, Beijing 100084, China
| |
Collapse
|