1
|
Pfeffer MA, Ling SSH, Wong JKW. Exploring the frontier: Transformer-based models in EEG signal analysis for brain-computer interfaces. Comput Biol Med 2024; 178:108705. [PMID: 38865781 DOI: 10.1016/j.compbiomed.2024.108705] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/11/2023] [Revised: 05/01/2024] [Accepted: 06/01/2024] [Indexed: 06/14/2024]
Abstract
This review systematically explores the application of transformer-based models in EEG signal processing and brain-computer interface (BCI) development, with a distinct focus on ensuring methodological rigour and adhering to empirical validations within the existing literature. By examining various transformer architectures, such as the Temporal Spatial Transformer Network (TSTN) and EEG Conformer, this review delineates their capabilities in mitigating challenges intrinsic to EEG data, such as noise and artifacts, and their subsequent implications on decoding and classification accuracies across disparate mental tasks. The analytical scope extends to a meticulous examination of attention mechanisms within transformer models, delineating their role in illuminating critical temporal and spatial EEG features and facilitating interpretability in model decision-making processes. The discourse additionally encapsulates emerging works that substantiate the efficacy of transformer models in noise reduction of EEG signals and diversifying applications beyond the conventional motor imagery paradigm. Furthermore, this review elucidates evident gaps and propounds exploratory avenues in the applications of pre-trained transformers in EEG analysis and the potential expansion into real-time and multi-task BCI applications. Collectively, this review distils extant knowledge, navigates through the empirical findings, and puts forward a structured synthesis, thereby serving as a conduit for informed future research endeavours in transformer-enhanced, EEG-based BCI systems.
Collapse
Affiliation(s)
- Maximilian Achim Pfeffer
- Faculty of Engineering and Information Technology, University of Technology Sydney, CB11 81-113, Broadway, Ultimo, 2007, New South Wales, Australia.
| | - Steve Sai Ho Ling
- Faculty of Engineering and Information Technology, University of Technology Sydney, CB11 81-113, Broadway, Ultimo, 2007, New South Wales, Australia.
| | - Johnny Kwok Wai Wong
- Faculty of Design, Architecture and Building, University of Technology Sydney, 15 Broadway, Ultimo, 2007, New South Wales, Australia.
| |
Collapse
|
2
|
Gouret A, Le Bars S, Porssut T, Waszak F, Chokron S. Advancements in brain-computer interfaces for the rehabilitation of unilateral spatial neglect: a concise review. Front Neurosci 2024; 18:1373377. [PMID: 38784094 PMCID: PMC11111994 DOI: 10.3389/fnins.2024.1373377] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/19/2024] [Accepted: 04/24/2024] [Indexed: 05/25/2024] Open
Abstract
This short review examines recent advancements in neurotechnologies within the context of managing unilateral spatial neglect (USN), a common condition following stroke. Despite the success of brain-computer interfaces (BCIs) in restoring motor function, there is a notable absence of effective BCI devices for treating cerebral visual impairments, a prevalent consequence of brain lesions that significantly hinders rehabilitation. This review analyzes current non-invasive BCIs and technological solutions dedicated to cognitive rehabilitation, with a focus on visuo-attentional disorders. We emphasize the need for further research into the use of BCIs for managing cognitive impairments and propose a new potential solution for USN rehabilitation, by combining the clinical subtleties of this syndrome with the technological advancements made in the field of neurotechnologies.
Collapse
Affiliation(s)
- Alix Gouret
- Integrative Neuroscience and Cognition Center (INCC), CNRS, Université Paris Cité, Paris, France
- Research and Innovation Department, Capgemini Engineering, Paris, France
| | - Solène Le Bars
- Integrative Neuroscience and Cognition Center (INCC), CNRS, Université Paris Cité, Paris, France
- Research and Innovation Department, Capgemini Engineering, Paris, France
| | - Thibault Porssut
- Research and Innovation Department, Capgemini Engineering, Paris, France
| | - Florian Waszak
- Integrative Neuroscience and Cognition Center (INCC), CNRS, Université Paris Cité, Paris, France
| | - Sylvie Chokron
- Integrative Neuroscience and Cognition Center (INCC), CNRS, Université Paris Cité, Paris, France
- Research and Innovation Department, Capgemini Engineering, Paris, France
| |
Collapse
|
3
|
Du X, Ding X, Xi M, Lv Y, Qiu S, Liu Q. A Data Augmentation Method for Motor Imagery EEG Signals Based on DCGAN-GP Network. Brain Sci 2024; 14:375. [PMID: 38672024 PMCID: PMC11048538 DOI: 10.3390/brainsci14040375] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/20/2024] [Revised: 04/09/2024] [Accepted: 04/11/2024] [Indexed: 04/28/2024] Open
Abstract
Motor imagery electroencephalography (EEG) signals have garnered attention in brain-computer interface (BCI) research due to their potential in promoting motor rehabilitation and control. However, the limited availability of labeled data poses challenges for training robust classifiers. In this study, we propose a novel data augmentation method utilizing an improved Deep Convolutional Generative Adversarial Network with Gradient Penalty (DCGAN-GP) to address this issue. We transformed raw EEG signals into two-dimensional time-frequency maps and employed a DCGAN-GP network to generate synthetic time-frequency representations resembling real data. Validation experiments were conducted on the BCI IV 2b dataset, comparing the performance of classifiers trained with augmented and unaugmented data. Results demonstrated that classifiers trained with synthetic data exhibit enhanced robustness across multiple subjects and achieve higher classification accuracy. Our findings highlight the effectiveness of utilizing a DCGAN-GP-generated synthetic EEG data to improve classifier performance in distinguishing different motor imagery tasks. Thus, the proposed data augmentation method based on a DCGAN-GP offers a promising avenue for enhancing BCI system performance, overcoming data scarcity challenges, and bolstering classifier robustness, thereby providing substantial support for the broader adoption of BCI technology in real-world applications.
Collapse
Affiliation(s)
| | - Xiaohui Ding
- Communication and Network Laboratory, Dalian University, Dalian 116622, China; (X.D.); (M.X.); (Y.L.); (S.Q.); (Q.L.)
| | | | | | | | | |
Collapse
|
4
|
Mahrooz MH, Fattahzadeh F, Gharibzadeh S. Decoding the Debate: A Comparative Study of Brain-Computer Interface and Neurofeedback. Appl Psychophysiol Biofeedback 2024; 49:47-53. [PMID: 37540396 DOI: 10.1007/s10484-023-09601-6] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Accepted: 07/25/2023] [Indexed: 08/05/2023]
Abstract
Brain-Computer Interface (BCI) and Neurofeedback (NF) both rely on the technology to capture brain activity. However, the literature lacks a clear distinction between the two, with some scholars categorizing NF as a special case of BCI while others view BCI as a natural extension of NF, or classify them as fundamentally different entities. This ambiguity hinders the flow of information and expertise among scholars and can cause confusion. To address this issue, we conducted a study comparing BCI and NF from two perspectives: the background and context within which BCI and NF developed, and their system design. We utilized Functional Flow Block Diagram (FFBD) as a system modelling approach to visualize inputs, functions, and outputs to compare BCI and NF at a conceptual level. Our analysis revealed that while NF is a subset of the biofeedback method that requires data from the brain to be extracted and processed, the device performing these tasks is a BCI system by definition. Therefore, we conclude that NF should be considered a specific application of BCI technology. By clarifying the relationship between BCI and NF, we hope to facilitate better communication and collaboration among scholars in these fields.
Collapse
Affiliation(s)
- Mohammad H Mahrooz
- Shahid Beheshti Medical University, Tehran, Iran.
- Department of aerospace engineering, Sharif University of Technology, Tehran, Iran.
| | | | - Shahriar Gharibzadeh
- Institue for cognitive and brain sciences, Shahid Beheshti University, Tehran, Iran
| |
Collapse
|
5
|
Nagarajan A, Robinson N, Ang KK, Chua KSG, Chew E, Guan C. Transferring a deep learning model from healthy subjects to stroke patients in a motor imagery brain-computer interface. J Neural Eng 2024; 21:016007. [PMID: 38091617 DOI: 10.1088/1741-2552/ad152f] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/19/2023] [Accepted: 12/13/2023] [Indexed: 01/18/2024]
Abstract
Objective.Motor imagery (MI) brain-computer interfaces (BCIs) based on electroencephalogram (EEG) have been developed primarily for stroke rehabilitation, however, due to limited stroke data, current deep learning methods for cross-subject classification rely on healthy data. This study aims to assess the feasibility of applying MI-BCI models pre-trained using data from healthy individuals to detect MI in stroke patients.Approach.We introduce a new transfer learning approach where features from two-class MI data of healthy individuals are used to detect MI in stroke patients. We compare the results of the proposed method with those obtained from analyses within stroke data. Experiments were conducted using Deep ConvNet and state-of-the-art subject-specific machine learning MI classifiers, evaluated on OpenBMI two-class MI-EEG data from healthy subjects and two-class MI versus rest data from stroke patients.Main results.Results of our study indicate that through domain adaptation of a model pre-trained using healthy subjects' data, an average MI detection accuracy of 71.15% (±12.46%) can be achieved across 71 stroke patients. We demonstrate that the accuracy of the pre-trained model increased by 18.15% after transfer learning (p<0.001). Additionally, the proposed transfer learning method outperforms the subject-specific results achieved by Deep ConvNet and FBCSP, with significant enhancements of 7.64% (p<0.001) and 5.55% (p<0.001) in performance, respectively. Notably, the healthy-to-stroke transfer learning approach achieved similar performance to stroke-to-stroke transfer learning, with no significant difference (p>0.05). Explainable AI analyses using transfer models determined channel relevance patterns that indicate contributions from the bilateral motor, frontal, and parietal regions of the cortex towards MI detection in stroke patients.Significance.Transfer learning from healthy to stroke can enhance the clinical use of BCI algorithms by overcoming the challenge of insufficient clinical data for optimal training.
Collapse
Affiliation(s)
- Aarthy Nagarajan
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
| | - Neethu Robinson
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
| | - Kai Keng Ang
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
- Institute for Infocomm Research, Agency of Science, Technology and Research (A*STAR), 1 Fusionopolis Way, Singapore 138632, Singapore
| | - Karen Sui Geok Chua
- Department of Rehabilitation Medicine, Tan Tock Seng Hospital, 11 Jln Tan Tock Seng, Singapore 308433, Singapore
| | - Effie Chew
- National University Health System, 1E Kent Ridge Road, Singapore 119228, Singapore
| | - Cuntai Guan
- School of Computer Science and Engineering, Nanyang Technological University, 50 Nanyang Ave, Singapore 639798, Singapore
| |
Collapse
|
6
|
Rodríguez-Azar PI, Mejía-Muñoz JM, Cruz-Mejía O, Torres-Escobar R, López LVR. Fog Computing for Control of Cyber-Physical Systems in Industry Using BCI. SENSORS (BASEL, SWITZERLAND) 2023; 24:149. [PMID: 38203012 PMCID: PMC10781321 DOI: 10.3390/s24010149] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/06/2023] [Revised: 12/23/2023] [Accepted: 12/25/2023] [Indexed: 01/12/2024]
Abstract
Brain-computer interfaces use signals from the brain, such as EEG, to determine brain states, which in turn can be used to issue commands, for example, to control industrial machinery. While Cloud computing can aid in the creation and operation of industrial multi-user BCI systems, the vast amount of data generated from EEG signals can lead to slow response time and bandwidth problems. Fog computing reduces latency in high-demand computation networks. Hence, this paper introduces a fog computing solution for BCI processing. The solution consists in using fog nodes that incorporate machine learning algorithms to convert EEG signals into commands to control a cyber-physical system. The machine learning module uses a deep learning encoder to generate feature images from EEG signals that are subsequently classified into commands by a random forest. The classification scheme is compared using various classifiers, being the random forest the one that obtained the best performance. Additionally, a comparison was made between the fog computing approach and using only cloud computing through the use of a fog computing simulator. The results indicate that the fog computing method resulted in less latency compared to the solely cloud computing approach.
Collapse
Affiliation(s)
- Paula Ivone Rodríguez-Azar
- Departamento de Ingeniería Industrial y Manufactura, Instituto de Ingeniería y Tecnología, Universidad Autónoma de Ciudad Juárez, Ciudad Juárez 32310, Mexico
| | - Jose Manuel Mejía-Muñoz
- Departamento de Ingeniería Eléctrica, Instituto de Ingenieria y Tecnologia, Universidad Autónoma de Ciudad Juárez, Ciudad Juárez 32310, Mexico;
| | - Oliverio Cruz-Mejía
- Departamento de Ingeniería Industrial, FES Aragón, Universidad Nacional Autónoma de México, Mexico 57171, Mexico;
| | | | - Lucero Verónica Ruelas López
- Departamento de Ingeniería Eléctrica, Instituto de Ingenieria y Tecnologia, Universidad Autónoma de Ciudad Juárez, Ciudad Juárez 32310, Mexico;
| |
Collapse
|
7
|
Maiseli B, Abdalla AT, Massawe LV, Mbise M, Mkocha K, Nassor NA, Ismail M, Michael J, Kimambo S. Brain-computer interface: trend, challenges, and threats. Brain Inform 2023; 10:20. [PMID: 37540385 PMCID: PMC10403483 DOI: 10.1186/s40708-023-00199-3] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/05/2023] [Accepted: 07/01/2023] [Indexed: 08/05/2023] Open
Abstract
Brain-computer interface (BCI), an emerging technology that facilitates communication between brain and computer, has attracted a great deal of research in recent years. Researchers provide experimental results demonstrating that BCI can restore the capabilities of physically challenged people, hence improving the quality of their lives. BCI has revolutionized and positively impacted several industries, including entertainment and gaming, automation and control, education, neuromarketing, and neuroergonomics. Notwithstanding its broad range of applications, the global trend of BCI remains lightly discussed in the literature. Understanding the trend may inform researchers and practitioners on the direction of the field, and on where they should invest their efforts more. Noting this significance, we have analyzed 25,336 metadata of BCI publications from Scopus to determine advancement of the field. The analysis shows an exponential growth of BCI publications in China from 2019 onwards, exceeding those from the United States that started to decline during the same period. Implications and reasons for this trend are discussed. Furthermore, we have extensively discussed challenges and threats limiting exploitation of BCI capabilities. A typical BCI architecture is hypothesized to address two prominent BCI threats, privacy and security, as an attempt to make the technology commercially viable to the society.
Collapse
Affiliation(s)
- Baraka Maiseli
- Department of Electronics and Telecommunications Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania.
| | - Abdi T Abdalla
- Department of Electronics and Telecommunications Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania
| | - Libe V Massawe
- Department of Electronics and Telecommunications Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania
| | - Mercy Mbise
- Department of Computer Science and Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania
| | - Khadija Mkocha
- Department of Electronics and Telecommunications Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania
| | - Nassor Ally Nassor
- Department of Electronics and Telecommunications Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania
| | - Moses Ismail
- Department of Electronics and Telecommunications Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania
| | - James Michael
- Department of Electronics and Telecommunications Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania
| | - Samwel Kimambo
- Department of Electronics and Telecommunications Engineering, College of Information and Communication Technologies, University of Dar es Salaam, 14113, Dar es Salaam, Tanzania
| |
Collapse
|
8
|
Pinheiro DJLL, Faber J, Micera S, Shokur S. Human-machine interface for two-dimensional steering control with the auricular muscles. Front Neurorobot 2023; 17:1154427. [PMID: 37342389 PMCID: PMC10277645 DOI: 10.3389/fnbot.2023.1154427] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/30/2023] [Accepted: 05/16/2023] [Indexed: 06/22/2023] Open
Abstract
Human-machine interfaces (HMIs) can be used to decode a user's motor intention to control an external device. People that suffer from motor disabilities, such as spinal cord injury, can benefit from the uses of these interfaces. While many solutions can be found in this direction, there is still room for improvement both from a decoding, hardware, and subject-motor learning perspective. Here we show, in a series of experiments with non-disabled participants, a novel decoding and training paradigm allowing naïve participants to use their auricular muscles (AM) to control two degrees of freedom with a virtual cursor. AMs are particularly interesting because they are vestigial muscles and are often preserved after neurological diseases. Our method relies on the use of surface electromyographic records and the use of contraction levels of both AMs to modulate the velocity and direction of a cursor in a two-dimensional paradigm. We used a locking mechanism to fix the current position of each axis separately to enable the user to stop the cursor at a certain location. A five-session training procedure (20-30 min per session) with a 2D center-out task was performed by five volunteers. All participants increased their success rate (Initial: 52.78 ± 5.56%; Final: 72.22 ± 6.67%; median ± median absolute deviation) and their trajectory performances throughout the training. We implemented a dual task with visual distractors to assess the mental challenge of controlling while executing another task; our results suggest that the participants could perform the task in cognitively demanding conditions (success rate of 66.67 ± 5.56%). Finally, using the Nasa Task Load Index questionnaire, we found that participants reported lower mental demand and effort in the last two sessions. To summarize, all subjects could learn to control the movement of a cursor with two degrees of freedom using their AM, with a low impact on the cognitive load. Our study is a first step in developing AM-based decoders for HMIs for people with motor disabilities, such as spinal cord injury.
Collapse
Affiliation(s)
- Daniel J. L. L. Pinheiro
- Division of Neuroscience, Department of Neurology and Neurosurgery, Neuroengineering and Neurocognition Laboratory, Escola Paulista de Medicina, Universidade Federal de São Paulo, São Paulo, Brazil
- Translational Neural Engineering Lab, Institute Neuro X, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| | - Jean Faber
- Division of Neuroscience, Department of Neurology and Neurosurgery, Neuroengineering and Neurocognition Laboratory, Escola Paulista de Medicina, Universidade Federal de São Paulo, São Paulo, Brazil
- Neuroengineering Laboratory, Division of Biomedical Engineering, Instituto de Ciência e Tecnologia, Universidade Federal de São Paulo, São José dos Campos, Brazil
| | - Silvestro Micera
- Translational Neural Engineering Lab, Institute Neuro X, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
- Department of Excellence in Robotics and AI, Institute of BioRobotics Interdisciplinary Health Center, Scuola Superiore Sant'Anna, Pisa, Italy
| | - Solaiman Shokur
- Translational Neural Engineering Lab, Institute Neuro X, École Polytechnique Fédérale de Lausanne, Geneva, Switzerland
| |
Collapse
|
9
|
Sciaraffa N, Di Flumeri G, Germano D, Giorgi A, Di Florio A, Borghini G, Vozzi A, Ronca V, Babiloni F, Aricò P. Evaluation of a New Lightweight EEG Technology for Translational Applications of Passive Brain-Computer Interfaces. Front Hum Neurosci 2022; 16:901387. [PMID: 35911603 PMCID: PMC9331459 DOI: 10.3389/fnhum.2022.901387] [Citation(s) in RCA: 10] [Impact Index Per Article: 5.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2022] [Accepted: 06/21/2022] [Indexed: 11/18/2022] Open
Abstract
Technologies like passive brain-computer interfaces (BCI) can enhance human-machine interaction. Anyhow, there are still shortcomings in terms of easiness of use, reliability, and generalizability that prevent passive-BCI from entering real-life situations. The current work aimed to technologically and methodologically design a new gel-free passive-BCI system for out-of-the-lab employment. The choice of the water-based electrodes and the design of a new lightweight headset met the need for easy-to-wear, comfortable, and highly acceptable technology. The proposed system showed high reliability in both laboratory and realistic settings, performing not significantly different from the gold standard based on gel electrodes. In both cases, the proposed system allowed effective discrimination (AUC > 0.9) between low and high levels of workload, vigilance, and stress even for high temporal resolution (<10 s). Finally, the generalizability of the proposed system has been tested through a cross-task calibration. The system calibrated with the data recorded during the laboratory tasks was able to discriminate the targeted human factors during the realistic task reaching AUC values higher than 0.8 at 40 s of temporal resolution in case of vigilance and workload, and 20 s of temporal resolution for the stress monitoring. These results pave the way for ecologic use of the system, where calibration data of the realistic task are difficult to obtain.
Collapse
Affiliation(s)
| | - Gianluca Di Flumeri
- BrainSigns Srl, Rome, Italy
- Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | | | - Andrea Giorgi
- BrainSigns Srl, Rome, Italy
- Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | | | - Gianluca Borghini
- BrainSigns Srl, Rome, Italy
- Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | - Alessia Vozzi
- BrainSigns Srl, Rome, Italy
- Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | - Vincenzo Ronca
- BrainSigns Srl, Rome, Italy
- Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
| | - Fabio Babiloni
- BrainSigns Srl, Rome, Italy
- Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
- College of Computer Science and Technology, Hangzhou Dianzi University, Hangzhou, China
| | - Pietro Aricò
- BrainSigns Srl, Rome, Italy
- Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy
- Department of Computer, Control, and Management Engineering “Antonio Ruberti”, Sapienza University of Rome, Rome, Italy
| |
Collapse
|
10
|
Al Boustani G, Weiß LJK, Li H, Meyer SM, Hiendlmeier L, Rinklin P, Menze B, Hemmert W, Wolfrum B. Influence of Auditory Cues on the Neuronal Response to Naturalistic Visual Stimuli in a Virtual Reality Setting. Front Hum Neurosci 2022; 16:809293. [PMID: 35721351 PMCID: PMC9201822 DOI: 10.3389/fnhum.2022.809293] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/04/2021] [Accepted: 05/02/2022] [Indexed: 11/13/2022] Open
Abstract
Virtual reality environments offer great opportunities to study the performance of brain-computer interfaces (BCIs) in real-world contexts. As real-world stimuli are typically multimodal, their neuronal integration elicits complex response patterns. To investigate the effect of additional auditory cues on the processing of visual information, we used virtual reality to mimic safety-related events in an industrial environment while we concomitantly recorded electroencephalography (EEG) signals. We simulated a box traveling on a conveyor belt system where two types of stimuli – an exploding and a burning box – interrupt regular operation. The recordings from 16 subjects were divided into two subsets, a visual-only and an audio-visual experiment. In the visual-only experiment, the response patterns for both stimuli elicited a similar pattern – a visual evoked potential (VEP) followed by an event-related potential (ERP) over the occipital-parietal lobe. Moreover, we found the perceived severity of the event to be reflected in the signal amplitude. Interestingly, the additional auditory cues had a twofold effect on the previous findings: The P1 component was significantly suppressed in the case of the exploding box stimulus, whereas the N2c showed an enhancement for the burning box stimulus. This result highlights the impact of multisensory integration on the performance of realistic BCI applications. Indeed, we observed alterations in the offline classification accuracy for a detection task based on a mixed feature extraction (variance, power spectral density, and discrete wavelet transform) and a support vector machine classifier. In the case of the explosion, the accuracy slightly decreased by –1.64% p. in an audio-visual experiment compared to the visual-only. Contrarily, the classification accuracy for the burning box increased by 5.58% p. when additional auditory cues were present. Hence, we conclude, that especially in challenging detection tasks, it is favorable to consider the potential of multisensory integration when BCIs are supposed to operate under (multimodal) real-world conditions.
Collapse
Affiliation(s)
- George Al Boustani
- Neuroelectronics – Munich Institute of Biomedical Engineering, Department of Electrical and Computer Engineering, Technical University of Munich, Munich, Germany
| | - Lennart Jakob Konstantin Weiß
- Neuroelectronics – Munich Institute of Biomedical Engineering, Department of Electrical and Computer Engineering, Technical University of Munich, Munich, Germany
| | - Hongwei Li
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
- Department of Informatics, Technical University of Munich, Munich, Germany
| | - Svea Marie Meyer
- Neuroelectronics – Munich Institute of Biomedical Engineering, Department of Electrical and Computer Engineering, Technical University of Munich, Munich, Germany
| | - Lukas Hiendlmeier
- Neuroelectronics – Munich Institute of Biomedical Engineering, Department of Electrical and Computer Engineering, Technical University of Munich, Munich, Germany
| | - Philipp Rinklin
- Neuroelectronics – Munich Institute of Biomedical Engineering, Department of Electrical and Computer Engineering, Technical University of Munich, Munich, Germany
| | - Bjoern Menze
- Department of Quantitative Biomedicine, University of Zurich, Zurich, Switzerland
- Department of Informatics, Technical University of Munich, Munich, Germany
| | - Werner Hemmert
- Bio-Inspired Information Processing – Munich Institute of Biomedical Engineering, Department of Electrical and Computer Engineering, Technical University of Munich, Munich, Germany
| | - Bernhard Wolfrum
- Neuroelectronics – Munich Institute of Biomedical Engineering, Department of Electrical and Computer Engineering, Technical University of Munich, Munich, Germany
- *Correspondence: Bernhard Wolfrum,
| |
Collapse
|
11
|
罗 建, 丁 鹏, 龚 安, 田 贵, 徐 浩, 赵 磊, 伏 云. [Applications, industrial transformation and commercial value of brain-computer interface technology]. SHENG WU YI XUE GONG CHENG XUE ZA ZHI = JOURNAL OF BIOMEDICAL ENGINEERING = SHENGWU YIXUE GONGCHENGXUE ZAZHI 2022; 39:405-415. [PMID: 35523563 PMCID: PMC9927342 DOI: 10.7507/1001-5515.202108068] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Subscribe] [Scholar Register] [Received: 08/22/2021] [Revised: 01/15/2022] [Indexed: 06/14/2023]
Abstract
Brain-computer interface (BCI) is a revolutionary human-computer interaction technology, which includes both BCI that can output instructions directly from the brain to external devices or machines without relying on the peripheral nerve and muscle system, and BCI that bypasses the peripheral nerve and muscle system and inputs electrical, magnetic, acoustic and optical stimuli or neural feedback directly to the brain from external devices or machines. With the development of BCI technology, it has potential application not only in medical field, but also in non-medical fields, such as education, military, finance, entertainment, smart home and so on. At present, there is little literature on the relevant application of BCI technology, the current situation of BCI industrialization at home and abroad and its commercial value. Therefore, this paper expounds and discusses the above contents, which are expected to provide valuable information for the public and organizations, BCI researchers, BCI industry translators and salespeople, and improve the cognitive level of BCI technology, further promote the application and industrial transformation of BCI technology and enhance the commercial value of BCI, so as to serve mankind better.
Collapse
Affiliation(s)
- 建功 罗
- 昆明理工大学 信息工程与自动化学院(昆明 650500)School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 鹏 丁
- 昆明理工大学 信息工程与自动化学院(昆明 650500)School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 安民 龚
- 昆明理工大学 信息工程与自动化学院(昆明 650500)School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 贵鑫 田
- 昆明理工大学 信息工程与自动化学院(昆明 650500)School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 浩天 徐
- 昆明理工大学 信息工程与自动化学院(昆明 650500)School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 磊 赵
- 昆明理工大学 信息工程与自动化学院(昆明 650500)School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, P. R. China
| | - 云发 伏
- 昆明理工大学 信息工程与自动化学院(昆明 650500)School of Information Engineering and Automation, Kunming University of Science and Technology, Kunming 650500, P. R. China
- 昆明理工大学 脑认知与脑机智能融合创新团队(昆明 650500)Brain Cognition and Brain-Computer Intelligence Integration Group, Kunming University of Science and Technology, Kunming 650500, P. R. China
| |
Collapse
|
12
|
Ascari L, Marchenkova A, Bellotti A, Lai S, Moro L, Koshmak K, Mantoan A, Barsotti M, Brondi R, Avveduto G, Sechi D, Compagno A, Avanzini P, Ambeck-Madsen J, Vecchiato G. Validation of a Novel Wearable Multistream Data Acquisition and Analysis System for Ergonomic Studies. SENSORS (BASEL, SWITZERLAND) 2021; 21:8167. [PMID: 34960261 PMCID: PMC8707223 DOI: 10.3390/s21248167] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 10/29/2021] [Revised: 11/30/2021] [Accepted: 12/03/2021] [Indexed: 12/02/2022]
Abstract
Nowadays, the growing interest in gathering physiological data and human behavior in everyday life scenarios is paralleled by an increase in wireless devices recording brain and body signals. However, the technical issues that characterize these solutions often limit the full brain-related assessments in real-life scenarios. Here we introduce the Biohub platform, a hardware/software (HW/SW) integrated wearable system for multistream synchronized acquisitions. This system consists of off-the-shelf hardware and state-of-art open-source software components, which are highly integrated into a high-tech low-cost solution, complete, yet easy to use outside conventional labs. It flexibly cooperates with several devices, regardless of the manufacturer, and overcomes the possibly limited resources of recording devices. The Biohub was validated through the characterization of the quality of (i) multistream synchronization, (ii) in-lab electroencephalographic (EEG) recordings compared with a medical-grade high-density device, and (iii) a Brain-Computer-Interface (BCI) in a real driving condition. Results show that this system can reliably acquire multiple data streams with high time accuracy and record standard quality EEG signals, becoming a valid device to be used for advanced ergonomics studies such as driving, telerehabilitation, and occupational safety.
Collapse
Affiliation(s)
- Luca Ascari
- Henesis s.r.l., 43123 Parma, Italy; (A.B.); (S.L.); (A.M.); (M.B.); (G.A.); (D.S.); (A.C.)
- Camlin Italy s.r.l., 43123 Parma, Italy; (L.M.); (K.K.); (R.B.)
| | - Anna Marchenkova
- Institute of Neuroscience, National Research Council of Italy, 43125 Parma, Italy; (A.M.); (P.A.)
| | - Andrea Bellotti
- Henesis s.r.l., 43123 Parma, Italy; (A.B.); (S.L.); (A.M.); (M.B.); (G.A.); (D.S.); (A.C.)
| | - Stefano Lai
- Henesis s.r.l., 43123 Parma, Italy; (A.B.); (S.L.); (A.M.); (M.B.); (G.A.); (D.S.); (A.C.)
| | - Lucia Moro
- Camlin Italy s.r.l., 43123 Parma, Italy; (L.M.); (K.K.); (R.B.)
| | | | - Alice Mantoan
- Henesis s.r.l., 43123 Parma, Italy; (A.B.); (S.L.); (A.M.); (M.B.); (G.A.); (D.S.); (A.C.)
| | - Michele Barsotti
- Henesis s.r.l., 43123 Parma, Italy; (A.B.); (S.L.); (A.M.); (M.B.); (G.A.); (D.S.); (A.C.)
| | | | - Giovanni Avveduto
- Henesis s.r.l., 43123 Parma, Italy; (A.B.); (S.L.); (A.M.); (M.B.); (G.A.); (D.S.); (A.C.)
| | - Davide Sechi
- Henesis s.r.l., 43123 Parma, Italy; (A.B.); (S.L.); (A.M.); (M.B.); (G.A.); (D.S.); (A.C.)
| | - Alberto Compagno
- Henesis s.r.l., 43123 Parma, Italy; (A.B.); (S.L.); (A.M.); (M.B.); (G.A.); (D.S.); (A.C.)
| | - Pietro Avanzini
- Institute of Neuroscience, National Research Council of Italy, 43125 Parma, Italy; (A.M.); (P.A.)
| | | | - Giovanni Vecchiato
- Institute of Neuroscience, National Research Council of Italy, 43125 Parma, Italy; (A.M.); (P.A.)
| |
Collapse
|
13
|
Vecchiato G. Hybrid Systems to Boost EEG-Based Real-Time Action Decoding in Car Driving Scenarios. FRONTIERS IN NEUROERGONOMICS 2021; 2:784827. [PMID: 38235223 PMCID: PMC10790909 DOI: 10.3389/fnrgo.2021.784827] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 09/28/2021] [Accepted: 11/08/2021] [Indexed: 01/19/2024]
Abstract
The complexity of concurrent cerebral processes underlying driving makes such human behavior one of the most studied real-world activities in neuroergonomics. Several attempts have been made to decode, both offline and online, cerebral activity during car driving with the ultimate goal to develop brain-based systems for assistive devices. Electroencephalography (EEG) is the cornerstone of these studies providing the highest temporal resolution to track those cerebral processes underlying overt behavior. Particularly when investigating real-world scenarios as driving, EEG is constrained by factors such as robustness, comfortability, and high data variability affecting the decoding performance. Hence, additional peripheral signals can be combined with EEG for increasing replicability and the overall performance of the brain-based action decoder. In this regard, hybrid systems have been proposed for the detection of braking and steering actions in driving scenarios to improve the predictive power of the single neurophysiological measurement. These recent results represent a proof of concept of the level of technological maturity. They may pave the way for increasing the predictive power of peripheral signals, such as electroculogram (EOG) and electromyography (EMG), collected in real-world scenarios when informed by EEG measurements, even if collected only offline in standard laboratory settings. The promising usability of such hybrid systems should be further investigated in other domains of neuroergonomics.
Collapse
Affiliation(s)
- Giovanni Vecchiato
- Institute of Neuroscience, National Research Council of Italy, Parma, Italy
| |
Collapse
|
14
|
Le Bars S, Chokron S, Balp R, Douibi K, Waszak F. Theoretical Perspective on an Ideomotor Brain-Computer Interface: Toward a Naturalistic and Non-invasive Brain-Computer Interface Paradigm Based on Action-Effect Representation. Front Hum Neurosci 2021; 15:732764. [PMID: 34776904 PMCID: PMC8581635 DOI: 10.3389/fnhum.2021.732764] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/29/2021] [Accepted: 10/11/2021] [Indexed: 11/13/2022] Open
Abstract
Recent years have been marked by the fulgurant expansion of non-invasive Brain-Computer Interface (BCI) devices and applications in various contexts (medical, industrial etc.). This technology allows agents "to directly act with thoughts," bypassing the peripheral motor system. Interestingly, it is worth noting that typical non-invasive BCI paradigms remain distant from neuroscientific models of human voluntary action. Notably, bidirectional links between action and perception are constantly ignored in BCI experiments. In the current perspective article, we proposed an innovative BCI paradigm that is directly inspired by the ideomotor principle, which postulates that voluntary actions are driven by the anticipated representation of forthcoming perceptual effects. We believe that (1) adapting BCI paradigms could allow simple action-effect bindings and consequently action-effect predictions and (2) using neural underpinnings of those action-effect predictions as features of interest in AI methods, could lead to more accurate and naturalistic BCI-mediated actions.
Collapse
Affiliation(s)
- Solène Le Bars
- Altran Lab, Capgemini Engineering, Paris, France.,Université de Paris, INCC UMR 8002, CNRS, Paris, France
| | - Sylvie Chokron
- Université de Paris, INCC UMR 8002, CNRS, Paris, France.,Fondation Ophtalmologique Adolphe de Rothschild, Paris, France
| | - Rodrigo Balp
- Altran Lab, Capgemini Engineering, Paris, France
| | | | | |
Collapse
|