1
|
Wang J, Lai Q, Han J, Qin P, Wu H. Neuroimaging biomarkers for the diagnosis and prognosis of patients with disorders of consciousness. Brain Res 2024; 1843:149133. [PMID: 39084451 DOI: 10.1016/j.brainres.2024.149133] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2023] [Revised: 05/29/2024] [Accepted: 07/25/2024] [Indexed: 08/02/2024]
Abstract
The progress in neuroimaging and electrophysiological techniques has shown substantial promise in improving the clinical assessment of disorders of consciousness (DOC). Through the examination of both stimulus-induced and spontaneous brain activity, numerous comprehensive investigations have explored variations in brain activity patterns among patients with DOC, yielding valuable insights for clinical diagnosis and prognostic purposes. Nonetheless, reaching a consensus on precise neuroimaging biomarkers for patients with DOC remains a challenge. Therefore, in this review, we begin by summarizing the empirical evidence related to neuroimaging biomarkers for DOC using various paradigms, including active, passive, and resting-state approaches, by employing task-based fMRI, resting-state fMRI (rs-fMRI), electroencephalography (EEG), and positron emission tomography (PET) techniques. Subsequently, we conducted a review of studies examining the neural correlates of consciousness in patients with DOC, with the findings holding potential value for the clinical application of DOC. Notably, previous research indicates that neuroimaging techniques have the potential to unveil covert awareness that conventional behavioral assessments might overlook. Furthermore, when integrated with various task paradigms or analytical approaches, this combination has the potential to significantly enhance the accuracy of both diagnosis and prognosis in DOC patients. Nonetheless, the stability of these neural biomarkers still needs additional validation, and future directions may entail integrating diagnostic and prognostic methods with big data and deep learning approaches.
Collapse
Affiliation(s)
- Jiaying Wang
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Center for Studies of Psychological Application, School of Psychology, South China Normal University, Guangzhou 510631, China
| | - Qiantu Lai
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Center for Studies of Psychological Application, School of Psychology, South China Normal University, Guangzhou 510631, China
| | - Junrong Han
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Institute for Brain Research and Rehabilitation, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, 510631 Guangzhou, China
| | - Pengmin Qin
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Center for Studies of Psychological Application, School of Psychology, South China Normal University, Guangzhou 510631, China; Pazhou Lab, Guangzhou 510330, China.
| | - Hang Wu
- Key Laboratory of Brain, Cognition and Education Sciences, Ministry of Education, Institute for Brain Research and Rehabilitation, and Guangdong Key Laboratory of Mental Health and Cognitive Science, South China Normal University, 510631 Guangzhou, China.
| |
Collapse
|
2
|
Edlow BL, Menon DK. Covert Consciousness in the ICU. Crit Care Med 2024; 52:1414-1426. [PMID: 39145701 DOI: 10.1097/ccm.0000000000006372] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 08/16/2024]
Abstract
OBJECTIVES For critically ill patients with acute severe brain injuries, consciousness may reemerge before behavioral responsiveness. The phenomenon of covert consciousness (i.e., cognitive motor dissociation) may be detected by advanced neurotechnologies such as task-based functional MRI (fMRI) and electroencephalography (EEG) in patients who appear unresponsive on the bedside behavioral examination. In this narrative review, we summarize the state-of-the-science in ICU detection of covert consciousness. Further, we consider the prognostic and therapeutic implications of diagnosing covert consciousness in the ICU, as well as its potential to inform discussions about continuation of life-sustaining therapy for patients with severe brain injuries. DATA SOURCES We reviewed salient medical literature regarding covert consciousness. STUDY SELECTION We included clinical studies investigating the diagnostic performance characteristics and prognostic utility of advanced neurotechnologies such as task-based fMRI and EEG. We focus on clinical guidelines, professional society scientific statements, and neuroethical analyses pertaining to the implementation of advanced neurotechnologies in the ICU to detect covert consciousness. DATA EXTRACTION AND DATA SYNTHESIS We extracted study results, guideline recommendations, and society scientific statement recommendations regarding the diagnostic, prognostic, and therapeutic relevance of covert consciousness to the clinical care of ICU patients with severe brain injuries. CONCLUSIONS Emerging evidence indicates that covert consciousness is present in approximately 15-20% of ICU patients who appear unresponsive on behavioral examination. Covert consciousness may be detected in patients with traumatic and nontraumatic brain injuries, including patients whose behavioral examination suggests a comatose state. The presence of covert consciousness in the ICU may predict the pace and extent of long-term functional recovery. Professional society guidelines now recommend assessment of covert consciousness using task-based fMRI and EEG. However, the clinical criteria for patient selection for such investigations are uncertain and global access to advanced neurotechnologies is limited.
Collapse
Affiliation(s)
- Brian L Edlow
- Center for Neurotechnology and Neurorecovery, Department of Neurology, Massachusetts General Hospital and Harvard Medical School, Boston, MA
- Athinoula A. Martinos Center for Biomedical Imaging, Massachusetts General Hospital, Charlestown, MA
| | - David K Menon
- University Division of Anaesthesia, University of Cambridge, Addenbrooke's Hospital Cambridge, Cambridge, United Kingdom
| |
Collapse
|
3
|
Qiu L, Zhong L, Li J, Feng W, Zhou C, Pan J. SFT-SGAT: A semi-supervised fine-tuning self-supervised graph attention network for emotion recognition and consciousness detection. Neural Netw 2024; 180:106643. [PMID: 39186838 DOI: 10.1016/j.neunet.2024.106643] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Revised: 04/11/2024] [Accepted: 08/14/2024] [Indexed: 08/28/2024]
Abstract
Emotional recognition is highly important in the field of brain-computer interfaces (BCIs). However, due to the individual variability in electroencephalogram (EEG) signals and the challenges in obtaining accurate emotional labels, traditional methods have shown poor performance in cross-subject emotion recognition. In this study, we propose a cross-subject EEG emotion recognition method based on a semi-supervised fine-tuning self-supervised graph attention network (SFT-SGAT). First, we model multi-channel EEG signals by constructing a graph structure that dynamically captures the spatiotemporal topological features of EEG signals. Second, we employ a self-supervised graph attention neural network to facilitate model training, mitigating the impact of signal noise on the model. Finally, a semi-supervised approach is used to fine-tune the model, enhancing its generalization ability in cross-subject classification. By combining supervised and unsupervised learning techniques, the SFT-SGAT maximizes the utility of limited labeled data in EEG emotion recognition tasks, thereby enhancing the model's performance. Experiments based on leave-one-subject-out cross-validation demonstrate that SFT-SGAT achieves state-of-the-art cross-subject emotion recognition performance on the SEED and SEED-IV datasets, with accuracies of 92.04% and 82.76%, respectively. Furthermore, experiments conducted on a self-collected dataset comprising ten healthy subjects and eight patients with disorders of consciousness (DOCs) revealed that the SFT-SGAT attains high classification performance in healthy subjects (maximum accuracy of 95.84%) and was successfully applied to DOC patients, with four patients achieving emotion recognition accuracies exceeding 60%. The experiments demonstrate the effectiveness of the proposed SFT-SGAT model in cross-subject EEG emotion recognition and its potential for assessing levels of consciousness in patients with DOC.
Collapse
Affiliation(s)
- Lina Qiu
- School of Artificial Intelligence, South China Normal University, Guangzhou, 510630, China; Research Station in Mathematics, South China Normal University, Guangzhou, 510630, China.
| | - Liangquan Zhong
- School of Artificial Intelligence, South China Normal University, Guangzhou, 510630, China.
| | - Jianping Li
- School of Artificial Intelligence, South China Normal University, Guangzhou, 510630, China.
| | - Weisen Feng
- School of Artificial Intelligence, South China Normal University, Guangzhou, 510630, China.
| | - Chengju Zhou
- School of Artificial Intelligence, South China Normal University, Guangzhou, 510630, China.
| | - Jiahui Pan
- School of Artificial Intelligence, South China Normal University, Guangzhou, 510630, China.
| |
Collapse
|
4
|
Pan J, Liang R, He Z, Li J, Liang Y, Zhou X, He Y, Li Y. ST-SCGNN: A Spatio-Temporal Self-Constructing Graph Neural Network for Cross-Subject EEG-Based Emotion Recognition and Consciousness Detection. IEEE J Biomed Health Inform 2024; 28:777-788. [PMID: 38015677 DOI: 10.1109/jbhi.2023.3335854] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/30/2023]
Abstract
In this paper, a novel spatio-temporal self-constructing graph neural network (ST-SCGNN) is proposed for cross-subject emotion recognition and consciousness detection. For spatio-temporal feature generation, activation and connection pattern features are first extracted and then combined to leverage their complementary emotion-related information. Next, a self-constructing graph neural network with a spatio-temporal model is presented. Specifically, the graph structure of the neural network is dynamically updated by the self-constructing module of the input signal. Experiments based on the SEED and SEED-IV datasets showed that the model achieved average accuracies of 85.90% and 76.37%, respectively. Both values exceed the state-of-the-art metrics with the same protocol. In clinical besides, patients with disorders of consciousness (DOC) suffer severe brain injuries, and sufficient training data for EEG-based emotion recognition cannot be collected. Our proposed ST-SCGNN method for cross-subject emotion recognition was first attempted in training in ten healthy subjects and testing in eight patients with DOC. We found that two patients obtained accuracies significantly higher than chance level and showed similar neural patterns with healthy subjects. Covert consciousness and emotion-related abilities were thus demonstrated in these two patients. Our proposed ST-SCGNN for cross-subject emotion recognition could be a promising tool for consciousness detection in DOC patients.
Collapse
|
5
|
Zhang J, Li J, Huang Z, Huang D, Yu H, Li Z. Recent Progress in Wearable Brain-Computer Interface (BCI) Devices Based on Electroencephalogram (EEG) for Medical Applications: A Review. HEALTH DATA SCIENCE 2023; 3:0096. [PMID: 38487198 PMCID: PMC10880169 DOI: 10.34133/hds.0096] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/24/2023] [Accepted: 10/19/2023] [Indexed: 03/17/2024]
Abstract
Importance: Brain-computer interface (BCI) decodes and converts brain signals into machine instructions to interoperate with the external world. However, limited by the implantation risks of invasive BCIs and the operational complexity of conventional noninvasive BCIs, applications of BCIs are mainly used in laboratory or clinical environments, which are not conducive to the daily use of BCI devices. With the increasing demand for intelligent medical care, the development of wearable BCI systems is necessary. Highlights: Based on the scalp-electroencephalogram (EEG), forehead-EEG, and ear-EEG, the state-of-the-art wearable BCI devices for disease management and patient assistance are reviewed. This paper focuses on the EEG acquisition equipment of the novel wearable BCI devices and summarizes the development direction of wearable EEG-based BCI devices. Conclusions: BCI devices play an essential role in the medical field. This review briefly summarizes novel wearable EEG-based BCIs applied in the medical field and the latest progress in related technologies, emphasizing its potential to help doctors, patients, and caregivers better understand and utilize BCI devices.
Collapse
Affiliation(s)
- Jiayan Zhang
- Beijing Advanced Innovation Center for Integrated Circuits, Beijing, China
- National Key Laboratory of Advanced Micro and Nano Manufacture Technology, School of Integrated Circuits,
Peking University, Beijing, China
| | - Junshi Li
- Beijing Advanced Innovation Center for Integrated Circuits, Beijing, China
- National Key Laboratory of Advanced Micro and Nano Manufacture Technology, School of Integrated Circuits,
Peking University, Beijing, China
| | - Zhe Huang
- Beijing Advanced Innovation Center for Integrated Circuits, Beijing, China
- National Key Laboratory of Advanced Micro and Nano Manufacture Technology, School of Integrated Circuits,
Peking University, Beijing, China
- Shenzhen Graduate School,
Peking University, Shenzhen, China
| | - Dong Huang
- Beijing Advanced Innovation Center for Integrated Circuits, Beijing, China
- National Key Laboratory of Advanced Micro and Nano Manufacture Technology, School of Integrated Circuits,
Peking University, Beijing, China
- School of Electronics,
Peking University, Beijing, China
| | - Huaiqiang Yu
- Sichuan Institute of Piezoelectric and Acousto-optic Technology, Chongqing, China
| | - Zhihong Li
- Beijing Advanced Innovation Center for Integrated Circuits, Beijing, China
- National Key Laboratory of Advanced Micro and Nano Manufacture Technology, School of Integrated Circuits,
Peking University, Beijing, China
| |
Collapse
|
6
|
Fu B, Gu C, Fu M, Xia Y, Liu Y. A novel feature fusion network for multimodal emotion recognition from EEG and eye movement signals. Front Neurosci 2023; 17:1234162. [PMID: 37600016 PMCID: PMC10436100 DOI: 10.3389/fnins.2023.1234162] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/03/2023] [Accepted: 07/20/2023] [Indexed: 08/22/2023] Open
Abstract
Emotion recognition is a challenging task, and the use of multimodal fusion methods for emotion recognition has become a trend. Fusion vectors can provide a more comprehensive representation of changes in the subject's emotional state, leading to more accurate emotion recognition results. Different fusion inputs or feature fusion methods have varying effects on the final fusion outcome. In this paper, we propose a novel Multimodal Feature Fusion Neural Network model (MFFNN) that effectively extracts complementary information from eye movement signals and performs feature fusion with EEG signals. We construct a dual-branch feature extraction module to extract features from both modalities while ensuring temporal alignment. A multi-scale feature fusion module is introduced, which utilizes cross-channel soft attention to adaptively select information from different spatial scales, enabling the acquisition of features at different spatial scales for effective fusion. We conduct experiments on the publicly available SEED-IV dataset, and our model achieves an accuracy of 87.32% in recognizing four emotions (happiness, sadness, fear, and neutrality). The results demonstrate that the proposed model can better explore complementary information from EEG and eye movement signals, thereby improving accuracy, and stability in emotion recognition.
Collapse
Affiliation(s)
- Baole Fu
- School of Automation, Qingdao University, Qingdao, China
- Institute for Future, Qingdao University, Qingdao, China
| | - Chunrui Gu
- School of Automation, Qingdao University, Qingdao, China
- Institute for Future, Qingdao University, Qingdao, China
| | - Ming Fu
- School of Automation, Qingdao University, Qingdao, China
- Institute for Future, Qingdao University, Qingdao, China
| | - Yuxiao Xia
- School of Automation, Qingdao University, Qingdao, China
- Institute for Future, Qingdao University, Qingdao, China
| | - Yinhua Liu
- School of Automation, Qingdao University, Qingdao, China
- Institute for Future, Qingdao University, Qingdao, China
- Shandong Key Laboratory of Industrial Control Technology, Qingdao, China
| |
Collapse
|
7
|
Gong L, Li M, Zhang T, Chen W. EEG emotion recognition using attention-based convolutional transformer neural network. Biomed Signal Process Control 2023. [DOI: 10.1016/j.bspc.2023.104835] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 03/12/2023]
|
8
|
He Q, He J, Yang Y, Zhao J. Brain-Computer Interfaces in Disorders of Consciousness. Neurosci Bull 2023; 39:348-352. [PMID: 35941403 PMCID: PMC9905465 DOI: 10.1007/s12264-022-00920-y] [Citation(s) in RCA: 1] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/18/2022] [Accepted: 06/03/2022] [Indexed: 11/25/2022] Open
Affiliation(s)
- Qiheng He
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, 100070, China
- China National Clinical Research Center for Neurological Diseases, Beijing, 100070, China
| | - Jianghong He
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, 100070, China.
| | - Yi Yang
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, 100070, China.
- Chinese Institute for Brain Research, Beijing, 100010, China.
- Beijing Institute of Brain Disorders, Beijing, 100069, China.
- China National Clinical Research Center for Neurological Diseases, Beijing, 100070, China.
| | - Jizong Zhao
- Department of Neurosurgery, Beijing Tiantan Hospital, Capital Medical University, Beijing, 100070, China
- China National Clinical Research Center for Neurological Diseases, Beijing, 100070, China
| |
Collapse
|
9
|
Galiotta V, Quattrociocchi I, D'Ippolito M, Schettini F, Aricò P, Sdoia S, Formisano R, Cincotti F, Mattia D, Riccio A. EEG-based Brain-Computer Interfaces for people with Disorders of Consciousness: Features and applications. A systematic review. Front Hum Neurosci 2022; 16:1040816. [PMID: 36545350 PMCID: PMC9760911 DOI: 10.3389/fnhum.2022.1040816] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/09/2022] [Accepted: 11/17/2022] [Indexed: 12/11/2022] Open
Abstract
Background Disorders of Consciousness (DoC) are clinical conditions following a severe acquired brain injury (ABI) characterized by absent or reduced awareness, known as coma, Vegetative State (VS)/Unresponsive Wakefulness Syndrome (VS/UWS), and Minimally Conscious State (MCS). Misdiagnosis rate between VS/UWS and MCS is attested around 40% due to the clinical and behavioral fluctuations of the patients during bedside consciousness assessments. Given the large body of evidence that some patients with DoC possess "covert" awareness, revealed by neuroimaging and neurophysiological techniques, they are candidates for intervention with brain-computer interfaces (BCIs). Objectives The aims of the present work are (i) to describe the characteristics of BCI systems based on electroencephalography (EEG) performed on DoC patients, in terms of control signals adopted to control the system, characteristics of the paradigm implemented, classification algorithms and applications (ii) to evaluate the performance of DoC patients with BCI. Methods The search was conducted on Pubmed, Web of Science, Scopus and Google Scholar. The PRISMA guidelines were followed in order to collect papers published in english, testing a BCI and including at least one DoC patient. Results Among the 527 papers identified with the first run of the search, 27 papers were included in the systematic review. Characteristics of the sample of participants, behavioral assessment, control signals employed to control the BCI, the classification algorithms, the characteristics of the paradigm, the applications and performance of BCI were the data extracted from the study. Control signals employed to operate the BCI were: P300 (N = 19), P300 and Steady-State Visual Evoked Potentials (SSVEP; hybrid system, N = 4), sensorimotor rhythms (SMRs; N = 5) and brain rhythms elicited by an emotional task (N = 1), while assessment, communication, prognosis, and rehabilitation were the possible applications of BCI in DoC patients. Conclusion Despite the BCI is a promising tool in the management of DoC patients, supporting diagnosis and prognosis evaluation, results are still preliminary, and no definitive conclusions may be drawn; even though neurophysiological methods, such as BCI, are more sensitive to covert cognition, it is suggested to adopt a multimodal approach and a repeated assessment strategy.
Collapse
Affiliation(s)
- Valentina Galiotta
- Neuroelectric Imaging and Brain-Computer Interface Laboratory, Fondazione Santa Lucia (IRCCS), Rome, Italy,Department of Psychology, Sapienza University of Rome, Rome, Italy
| | - Ilaria Quattrociocchi
- Neuroelectric Imaging and Brain-Computer Interface Laboratory, Fondazione Santa Lucia (IRCCS), Rome, Italy,Department of Computer, Control, and Management Engineering “Antonio Ruberti”, Sapienza University of Rome, Rome, Italy
| | - Mariagrazia D'Ippolito
- Neuroelectric Imaging and Brain-Computer Interface Laboratory, Fondazione Santa Lucia (IRCCS), Rome, Italy,*Correspondence: Mariagrazia D'Ippolito
| | - Francesca Schettini
- Neuroelectric Imaging and Brain-Computer Interface Laboratory, Fondazione Santa Lucia (IRCCS), Rome, Italy,Servizio di Ausilioteca per la Riabilitazione Assistita con Tecnologia, Fondazione Santa Lucia (IRCCS), Rome, Italy
| | - Pietro Aricò
- Department of Computer, Control, and Management Engineering “Antonio Ruberti”, Sapienza University of Rome, Rome, Italy,Department of Molecular Medicine, Sapienza University of Rome, Rome, Italy,BrainSigns srl, Rome, Italy
| | - Stefano Sdoia
- Department of Psychology, Sapienza University of Rome, Rome, Italy
| | - Rita Formisano
- Neurorehabilitation 2 and Post-Coma Unit, Fondazione Santa Lucia (IRCCS), Rome, Italy
| | - Febo Cincotti
- Department of Computer, Control, and Management Engineering “Antonio Ruberti”, Sapienza University of Rome, Rome, Italy
| | - Donatella Mattia
- Neuroelectric Imaging and Brain-Computer Interface Laboratory, Fondazione Santa Lucia (IRCCS), Rome, Italy,Servizio di Ausilioteca per la Riabilitazione Assistita con Tecnologia, Fondazione Santa Lucia (IRCCS), Rome, Italy
| | - Angela Riccio
- Neuroelectric Imaging and Brain-Computer Interface Laboratory, Fondazione Santa Lucia (IRCCS), Rome, Italy,Servizio di Ausilioteca per la Riabilitazione Assistita con Tecnologia, Fondazione Santa Lucia (IRCCS), Rome, Italy
| |
Collapse
|
10
|
Qu T, Jin J, Xu R, Wang X, Cichocki A. Riemannian distance based channel selection and feature extraction combining discriminative time-frequency bands and Riemannian tangent space for MI-BCIs. J Neural Eng 2022; 19. [PMID: 36126643 DOI: 10.1088/1741-2552/ac9338] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/22/2022] [Accepted: 09/20/2022] [Indexed: 11/12/2022]
Abstract
OBJECTIVE Motor imagery-based brain computer interfaces (MI-BCIs) have been widely researched because they do not demand external stimuli and have a high degree of maneuverability. In most scenarios, superabundant selected channels, fixed time windows, and frequency bands would certainly affect the performance of MI-BCIs due to the neurophysiological diversities between different individuals. In this study, we tended to effectively use the Riemannian geometry of spatial covariance matrix to extract more robust features and thus enhance the decoding efficiency. APPROACH First, we propose a Riemannian distance-based EEG channel selection method (RDCS), which preliminarily reduces the information redundancy in the first stage. Second, we extract discriminative Riemannian Tangent Space features of EEG signals of selected channels from the most discriminant time-frequency bands (DTFRTS) to further enhance decoding accuracy for MI-BCIs. Finally, we trained a support vector machine (SVM) model with a linear kernel to classify our extracted discriminative Riemannian features and evaluated our proposed method using publicly available BCI Competition Ⅳ dataset Ⅰ (DS1) and Competition Ⅲ dataset Ⅲa (DS2). MAIN RESULTS The experimental results showed that the average classification accuracy with the selected 10-channel EEG signals of our method is 88.1% and 91.6% in DS1 and DS2 respectively. The average improvements are 24.3% & 27.1% on DS1 and 4.4% & 14.2% on DS2 for 10 & 20 selected channels, respectively. SIGNIFICANCE These results showed that our proposed method is a promising candidate for performance improvement of MI-BCIs.
Collapse
Affiliation(s)
- Tingnan Qu
- Key Laboratory of Advanced Control and Optimization for Chemical Processes Ministry of Education, East China University of Science and Technology, 130 Meilong Road, Xuhui District, Shanghai, 200237, CHINA
| | - Jing Jin
- Key Laboratory of Advanced Control and Optimization for Chemical Processes Ministry of Education, East China University of Science and Technology, 130 Meilong Road, Xuhui District, Shanghai 200237, Shanghai, Shanghai, 200237, CHINA
| | - Ren Xu
- Guger Technologies OG, Research and Software Developmentg.tec - Guger Technologies Sierningstrasse 14, 4521 Schiedlberg, Graz, 8020, AUSTRIA
| | - Xingyu Wang
- Key Laboratory of Advanced Control and Optimization for Chemical Processes Ministry of Education, East China University of Science and Technology, 130 Meilong Road, Shanghai, Shanghai, 200237, CHINA
| | - Andrzej Cichocki
- Skolkovo Institute of Science and Technology, Bolshoy Boulevard 30, bld. 1 Moscow, Russia 121205, Skolkovo, Moskovskaâ, 121205, RUSSIAN FEDERATION
| |
Collapse
|
11
|
Spataro R, Xu Y, Xu R, Mandalà G, Allison BZ, Ortner R, Heilinger A, La Bella V, Guger C. How brain-computer interface technology may improve the diagnosis of the disorders of consciousness: A comparative study. Front Neurosci 2022; 16:959339. [PMID: 36033632 PMCID: PMC9404379 DOI: 10.3389/fnins.2022.959339] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/01/2022] [Accepted: 07/18/2022] [Indexed: 01/18/2023] Open
Abstract
Objective Clinical assessment of consciousness relies on behavioural assessments, which have several limitations. Hence, disorder of consciousness (DOC) patients are often misdiagnosed. In this work, we aimed to compare the repetitive assessment of consciousness performed with a clinical behavioural and a Brain-Computer Interface (BCI) approach. Materials and methods For 7 weeks, sixteen DOC patients participated in weekly evaluations using both the Coma Recovery Scale-Revised (CRS-R) and a vibrotactile P300 BCI paradigm. To use the BCI, patients had to perform an active mental task that required detecting specific stimuli while ignoring other stimuli. We analysed the reliability and the efficacy in the detection of command following resulting from the two methodologies. Results Over repetitive administrations, the BCI paradigm detected command following before the CRS-R in seven patients. Four clinically unresponsive patients consistently showed command following during the BCI assessments. Conclusion Brain-Computer Interface active paradigms might contribute to the evaluation of the level of consciousness, increasing the diagnostic precision of the clinical bedside approach. Significance The integration of different diagnostic methods leads to a better knowledge and care for the DOC.
Collapse
Affiliation(s)
- Rossella Spataro
- IRCCS Centro Neurolesi Bonino Pulejo, Palermo, Italy
- ALS Clinical Research Center, University of Palermo, Palermo, Italy
- *Correspondence: Rossella Spataro,
| | - Yiyan Xu
- ALS Clinical Research Center, University of Palermo, Palermo, Italy
| | - Ren Xu
- g.tec Medical Engineering GmbH, Schiedlberg, Austria
| | - Giorgio Mandalà
- Rehabilitation Unit, Ospedale Buccheri La Ferla, Palermo, Italy
| | - Brendan Z. Allison
- Cognitive Science Department, University of California, San Diego, San Diego, United States
| | - Rupert Ortner
- g.tec Medical Engineering Spain S.L., Barcelona, Spain
| | | | | | - Christoph Guger
- g.tec Medical Engineering GmbH, Schiedlberg, Austria
- g.tec Medical Engineering Spain S.L., Barcelona, Spain
| |
Collapse
|
12
|
Aydın S, Akın B. Machine learning classification of maladaptive rumination and cognitive distraction in terms of frequency specific complexity. Biomed Signal Process Control 2022. [DOI: 10.1016/j.bspc.2022.103740] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/02/2022]
|
13
|
Fusion of EEG-Based Activation, Spatial, and Connection Patterns for Fear Emotion Recognition. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2022; 2022:3854513. [PMID: 35463262 PMCID: PMC9020909 DOI: 10.1155/2022/3854513] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 11/10/2021] [Accepted: 03/19/2022] [Indexed: 11/29/2022]
Abstract
At present, emotion recognition based on electroencephalograms (EEGs) has attracted much more attention. Current studies of affective brain-computer interfaces (BCIs) focus on the recognition of happiness and sadness using brain activation patterns. Fear recognition involving brain activities in different spatial distributions and different brain functional networks has been scarcely investigated. In this study, we propose a multifeature fusion method combining energy activation, spatial distribution, and brain functional connection network (BFCN) features for fear emotion recognition. The affective brain pattern was identified by not only the power activation features of differential entropy (DE) but also the spatial distribution features of the common spatial pattern (CSP) and the EEG phase synchronization features of phase lock value (PLV). A total of 15 healthy subjects took part in the experiment, and the average accuracy rate was 85.00% ± 8.13%. The experimental results showed that the fear emotions of subjects were fully stimulated and effectively identified. The proposed fusion method on fear recognition was thus validated and is of great significance to the development of effective emotional BCI systems.
Collapse
|
14
|
Värbu K, Muhammad N, Muhammad Y. Past, Present, and Future of EEG-Based BCI Applications. SENSORS (BASEL, SWITZERLAND) 2022; 22:3331. [PMID: 35591021 PMCID: PMC9101004 DOI: 10.3390/s22093331] [Citation(s) in RCA: 14] [Impact Index Per Article: 7.0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 02/23/2022] [Revised: 04/05/2022] [Accepted: 04/25/2022] [Indexed: 06/15/2023]
Abstract
An electroencephalography (EEG)-based brain-computer interface (BCI) is a system that provides a pathway between the brain and external devices by interpreting EEG. EEG-based BCI applications have initially been developed for medical purposes, with the aim of facilitating the return of patients to normal life. In addition to the initial aim, EEG-based BCI applications have also gained increasing significance in the non-medical domain, improving the life of healthy people, for instance, by making it more efficient, collaborative and helping develop themselves. The objective of this review is to give a systematic overview of the literature on EEG-based BCI applications from the period of 2009 until 2019. The systematic literature review has been prepared based on three databases PubMed, Web of Science and Scopus. This review was conducted following the PRISMA model. In this review, 202 publications were selected based on specific eligibility criteria. The distribution of the research between the medical and non-medical domain has been analyzed and further categorized into fields of research within the reviewed domains. In this review, the equipment used for gathering EEG data and signal processing methods have also been reviewed. Additionally, current challenges in the field and possibilities for the future have been analyzed.
Collapse
Affiliation(s)
- Kaido Värbu
- Institute of Computer Science, University of Tartu, 51009 Tartu, Estonia;
| | - Naveed Muhammad
- Institute of Computer Science, University of Tartu, 51009 Tartu, Estonia;
| | - Yar Muhammad
- Department of Computing & Games, School of Computing, Engineering & Digital Technologies, Teesside University, Middlesbrough TS1 3BX, UK;
| |
Collapse
|
15
|
Objectivity meets subjectivity: A subjective and objective feature fused neural network for emotion recognition. Appl Soft Comput 2022. [DOI: 10.1016/j.asoc.2022.108889] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/18/2022]
|
16
|
Qiu L, Zhong Y, Xie Q, He Z, Wang X, Chen Y, Zhan CA, Pan J. Multi-Modal Integration of EEG-fNIRS for Characterization of Brain Activity Evoked by Preferred Music. Front Neurorobot 2022; 16:823435. [PMID: 35173597 PMCID: PMC8841473 DOI: 10.3389/fnbot.2022.823435] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/27/2021] [Accepted: 01/07/2022] [Indexed: 11/20/2022] Open
Abstract
Music can effectively improve people's emotions, and has now become an effective auxiliary treatment method in modern medicine. With the rapid development of neuroimaging, the relationship between music and brain function has attracted much attention. In this study, we proposed an integrated framework of multi-modal electroencephalogram (EEG) and functional near infrared spectroscopy (fNIRS) from data collection to data analysis to explore the effects of music (especially personal preferred music) on brain activity. During the experiment, each subject was listening to two different kinds of music, namely personal preferred music and neutral music. In analyzing the synchronization signals of EEG and fNIRS, we found that music promotes the activity of the brain (especially the prefrontal lobe), and the activation induced by preferred music is stronger than that of neutral music. For the multi-modal features of EEG and fNIRS, we proposed an improved Normalized-ReliefF method to fuse and optimize them and found that it can effectively improve the accuracy of distinguishing between the brain activity evoked by preferred music and neutral music (up to 98.38%). Our work provides an objective reference based on neuroimaging for the research and application of personalized music therapy.
Collapse
Affiliation(s)
- Lina Qiu
- School of Software, South China Normal University, Guangzhou, China
| | - Yongshi Zhong
- School of Software, South China Normal University, Guangzhou, China
| | - Qiuyou Xie
- Department of Rehabilitation Medicine, Zhujiang Hospital, Southern Medical University, Guangzhou, China
| | - Zhipeng He
- School of Software, South China Normal University, Guangzhou, China
| | - Xiaoyun Wang
- Guangdong Work Injury Rehabilitation Hospital, Guangzhou, China
| | - Yingyue Chen
- Guangdong Work Injury Rehabilitation Hospital, Guangzhou, China
| | - Chang'an A. Zhan
- School of Biomedical Engineering, Southern Medical University, Guangzhou, China
- Chang'an A. Zhan
| | - Jiahui Pan
- School of Software, South China Normal University, Guangzhou, China
- *Correspondence: Jiahui Pan
| |
Collapse
|
17
|
Shankhdhar A, Verma PK, Agrawal P, Madaan V, Gupta C. Quality analysis for reliable complex multiclass neuroscience signal classification via electroencephalography. INTERNATIONAL JOURNAL OF QUALITY & RELIABILITY MANAGEMENT 2022. [DOI: 10.1108/ijqrm-07-2021-0237] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
PurposeThe aim of this paper is to explore the brain–computer interface (BCI) as a methodology for generating awareness and increasing reliable use cases of the same so that an individual's quality of life can be enhanced via neuroscience and neural networks, and risk evaluation of certain experiments of BCI can be conducted in a proactive manner.Design/methodology/approachThis paper puts forward an efficient approach for an existing BCI device, which can enhance the performance of an electroencephalography (EEG) signal classifier in a composite multiclass problem and investigates the effects of sampling rate on feature extraction and multiple channels on the accuracy of a complex multiclass EEG signal. A one-dimensional convolutional neural network architecture is used to further classify and improve the quality of the EEG signals, and other algorithms are applied to test their variability. The paper further also dwells upon the combination of internet of things multimedia technology to be integrated with a customized design BCI network based on a conventionally used system known as the message query telemetry transport.FindingsAt the end of our implementation stage, 98% accuracy was achieved in a binary classification problem of classifying digit and non-digit stimuli, and 36% accuracy was observed in the classification of signals resulting from stimuli of digits 0 to 9.Originality/valueBCI, also known as the neural-control interface, is a device that helps a user reliably interact with a computer using only his/her brain activity, which is measured usually via EEG. An EEG machine is a quality device used for observing the neural activity and electric signals generated in certain parts of the human brain, which in turn can help us in studying the different core components of the human brain and how it functions to improve the quality of human life in general.
Collapse
|
18
|
Graham M. Residual Cognitive Capacities in Patients With Cognitive Motor Dissociation, and Their Implications for Well-Being. THE JOURNAL OF MEDICINE AND PHILOSOPHY 2021; 46:729-757. [PMID: 34655220 PMCID: PMC8643594 DOI: 10.1093/jmp/jhab026] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/18/2023] Open
Abstract
Patients with severe disorders of consciousness are thought to be unaware of themselves or their environment. However, research suggests that a minority of patients diagnosed as having a disorder of consciousness remain aware. These patients, designated as having “cognitive motor dissociation” (CMD), can demonstrate awareness by imagining specific tasks, which generates brain activity detectable via functional neuroimaging. The discovery of consciousness in these patients raises difficult questions about their well-being, and it has been argued that it would be better for these patients if they were allowed to die. Conversely, I argue that CMD patients may have a much higher level of well-being than is generally acknowledged. It is far from clear that their lives are not worth living, because there are still significant gaps in our understanding of how these patients experience the world. I attempt to fill these gaps, by analyzing the neuroscientific research that has taken place with these patients to date. Having generated as comprehensive a picture as possible of the capacities of CMD patients, I examine this picture through the lens of traditional philosophical theories of well-being. I conclude that the presumption that CMD patients do not have lives worth living is not adequately supported.
Collapse
|
19
|
He Z, Zhong Y, Pan J. An adversarial discriminative temporal convolutional network for EEG-based cross-domain emotion recognition. Comput Biol Med 2021; 141:105048. [PMID: 34838262 DOI: 10.1016/j.compbiomed.2021.105048] [Citation(s) in RCA: 18] [Impact Index Per Article: 6.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/16/2021] [Revised: 11/15/2021] [Accepted: 11/15/2021] [Indexed: 11/16/2022]
Abstract
Domain adaptation (DA) tackles the problem where data from the source domain and target domain have different underlying distributions. In cross-domain (cross-subject or cross-dataset) emotion recognition based on EEG signals, traditional classification methods lack domain adaptation capabilities and have low performance. To address this problem, we proposed a novel domain adaptation strategy called adversarial discriminative-temporal convolutional networks (AD-TCNs) in this study, which can ensure the invariance of the representation of feature graphs in different domains and fill in the differences between different domains. For EEG data with specific temporal attributes, the temporal model TCN is used as the feature encoder. In the cross-subject experiment, our AD-TCN method achieved the highest accuracies of the valence and arousal dimensions in both the DREAMER and DEAP datasets. In the cross-dataset experiment, two of the eight task groups showed accuracies of 62.65% and 62.36%. Compared with the state-of-the-art performance in the same protocol, experimental results demonstrated that our method is an effective extension to realize EEG-based cross-domain emotion recognition.
Collapse
Affiliation(s)
- Zhipeng He
- School of Software, South China Normal University, Guangzhou, 510631, China
| | - Yongshi Zhong
- School of Software, South China Normal University, Guangzhou, 510631, China
| | - Jiahui Pan
- School of Software, South China Normal University, Guangzhou, 510631, China; Pazhou Lab, Guangzhou, 510330, China.
| |
Collapse
|
20
|
Yan W, Liu X, Shan B, Zhang X, Pu Y. Research on the Emotions Based on Brain -Computer Technology: A Bibliometric Analysis and Research Agenda. Front Psychol 2021; 12:771591. [PMID: 34790157 PMCID: PMC8591067 DOI: 10.3389/fpsyg.2021.771591] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/06/2021] [Accepted: 10/07/2021] [Indexed: 11/13/2022] Open
Abstract
This study conducts a scientific analysis of 249 literature on the application of brain-computer technology in emotion research. We find that existing researches mainly focus on engineering, computer science, neurosciences neurology and psychology. PR China, United States, and Germany have the largest number of publications. Authors can be divided into four groups: real-time functional magnetic resonance imaging (rtfMRI) research group, brain-computer interface (BCI) impact factors analysis group, brain-computer music interfacing (BCMI) group, and user status research group. Clustering results can be divided into five categories, including external stimulus and event-related potential (ERP), electroencephalography (EEG), and information collection, support vector machine (SVM) and information processing, deep learning and emotion recognition, neurofeedback, and self-regulation. Based on prior researches, this study points out that individual differences, privacy risk, the extended study of BCI application scenarios and others deserve further research.
Collapse
Affiliation(s)
- Wei Yan
- School of Management, Jilin University, Changchun, China
| | - Xiaoju Liu
- School of Management, Jilin University, Changchun, China
| | - Biaoan Shan
- School of Management, Jilin University, Changchun, China
| | | | - Yi Pu
- School of Management, Jilin University, Changchun, China
| |
Collapse
|
21
|
Song X, Zeng Y, Tong L, Shu J, Bao G, Yan B. P3-MSDA: Multi-Source Domain Adaptation Network for Dynamic Visual Target Detection. Front Hum Neurosci 2021; 15:685173. [PMID: 34434096 PMCID: PMC8381600 DOI: 10.3389/fnhum.2021.685173] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/24/2021] [Accepted: 07/07/2021] [Indexed: 11/13/2022] Open
Abstract
Single-trial electroencephalogram detection has been widely applied in brain-computer interface (BCI) systems. Moreover, an individual generalized model is significant for applying the dynamic visual target detection BCI system in real life because of the time jitter of the detection latency, the dynamics and complexity of visual background. Hence, we developed an unsupervised multi-source domain adaptation network (P3-MSDA) for dynamic visual target detection. In this network, a P3 map-clustering method was proposed for source domain selection. The adversarial domain adaptation was conducted for domain alignment to eliminate individual differences, and prediction probabilities were ranked and returned to guide the input of target samples for imbalanced data classification. The results showed that individuals with a strong P3 map selected by the proposed P3 map-clustering method perform best on the source domain. Compared with existing schemes, the proposed P3-MSDA network achieved the highest classification accuracy and F1 score using five labeled individuals with a strong P3 map as the source domain. These findings can have a significant meaning in building an individual generalized model for dynamic visual target detection.
Collapse
Affiliation(s)
- Xiyu Song
- Henan Key Laboratory of Imaging and Intelligent Processing, Chinese People's Liberation Army (PLA) Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Ying Zeng
- Henan Key Laboratory of Imaging and Intelligent Processing, Chinese People's Liberation Army (PLA) Strategic Support Force Information Engineering University, Zhengzhou, China.,Key Laboratory for NeuroInformation of Ministry of Education, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu, China
| | - Li Tong
- Henan Key Laboratory of Imaging and Intelligent Processing, Chinese People's Liberation Army (PLA) Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Jun Shu
- Henan Key Laboratory of Imaging and Intelligent Processing, Chinese People's Liberation Army (PLA) Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Guangcheng Bao
- Henan Key Laboratory of Imaging and Intelligent Processing, Chinese People's Liberation Army (PLA) Strategic Support Force Information Engineering University, Zhengzhou, China
| | - Bin Yan
- Henan Key Laboratory of Imaging and Intelligent Processing, Chinese People's Liberation Army (PLA) Strategic Support Force Information Engineering University, Zhengzhou, China
| |
Collapse
|
22
|
Salama ES, El-Khoribi RA, Shoman ME, Wahby Shalaby MA. A 3D-convolutional neural network framework with ensemble learning techniques for multi-modal emotion recognition. EGYPTIAN INFORMATICS JOURNAL 2021. [DOI: 10.1016/j.eij.2020.07.005] [Citation(s) in RCA: 8] [Impact Index Per Article: 2.7] [Reference Citation Analysis] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 10/23/2022]
|
23
|
Schmitz S. TechnoBrainBodies-in-Cultures: An Intersectional Case. FRONTIERS IN SOCIOLOGY 2021; 6:651486. [PMID: 33987221 PMCID: PMC8112819 DOI: 10.3389/fsoc.2021.651486] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 01/09/2021] [Accepted: 02/23/2021] [Indexed: 06/12/2023]
Abstract
The cyborgization of brainbodies with computer hardware and software today ranges in scope from the realization of Brain-Computer Interfaces (BCIs) to visions of mind upload to silicon, the latter being targeted toward a transhuman future. Refining posthumanist concepts to formulate a posthumanities perspective, and contrasting those approaches with transhumanist trajectories, I explore the intersectional dimension of realizations and visions of neuro-technological developments, which I name TechnoBrainBodies-in-Cultures. In an intersectional analysis, I investigate the embedding and legitimation of transhumanist visions brought about by neuroscientific research and neuro-technological development based on a concept of modern neurobiological determinism. The conjoined trajectories of BCI research and development and transhumanist visions perpetuate the inscription of intersectional norms, with the concomitant danger of producing discriminatory effects. This culminates in normative capacity being seen as a conflation of the abled, successful, white masculinized techno-brain with competition. My deeper analysis, however, also enables displacements within recent BCI research and development to be characterized: from ''thought-translation" to affective conditioning and from controllability to obstinacy within the BCI, going so far as to open the closed loop. These realizations challenge notions about the BCI's actor status and agency and foster questions about shifts in the corresponding subject-object relations. Based on these analyses, I look at the effects of neuro-technological and transhumanist governmentality on the question of whose lives are to be improved and whose lives should be excluded from these developments. Within the framework of political feminist materialisms, I combine the concept of posthumanities with my concept of TechnoBrainBodies-in-Cultures to envision and discuss a material-discursive strategy, encompassing dimensions of affect, sociality, resistance, compassion, cultural diversity, ethnic diversity, multiple sexes/sexualities, aging, dis/abilities-in short, all of this "intersectional stuff"-as well as obstinate techno-brain agencies and contumacies foreseen in these cyborgian futures.
Collapse
|
24
|
Keshmiri S. Conditional Entropy: A Potential Digital Marker for Stress. ENTROPY (BASEL, SWITZERLAND) 2021; 23:286. [PMID: 33652891 PMCID: PMC7996836 DOI: 10.3390/e23030286] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 01/27/2021] [Revised: 02/20/2021] [Accepted: 02/23/2021] [Indexed: 12/12/2022]
Abstract
Recent decades have witnessed a substantial progress in the utilization of brain activity for the identification of stress digital markers. In particular, the success of entropic measures for this purpose is very appealing, considering (1) their suitability for capturing both linear and non-linear characteristics of brain activity recordings and (2) their direct association with the brain signal variability. These findings rely on external stimuli to induce the brain stress response. On the other hand, research suggests that the use of different types of experimentally induced psychological and physical stressors could potentially yield differential impacts on the brain response to stress and therefore should be dissociated from more general patterns. The present study takes a step toward addressing this issue by introducing conditional entropy (CE) as a potential electroencephalography (EEG)-based resting-state digital marker of stress. For this purpose, we use the resting-state multi-channel EEG recordings of 20 individuals whose responses to stress-related questionnaires show significantly higher and lower level of stress. Through the application of representational similarity analysis (RSA) and K-nearest-neighbor (KNN) classification, we verify the potential that the use of CE can offer to the solution concept of finding an effective digital marker for stress.
Collapse
Affiliation(s)
- Soheil Keshmiri
- Advanced Telecommunications Research Institute International (ATR), Kyoto 619-0237, Japan
| |
Collapse
|
25
|
Evaluation of Emotional Satisfaction Using Questionnaires in Voice-Based Human–AI Interaction. APPLIED SCIENCES-BASEL 2021. [DOI: 10.3390/app11041920] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
With the development of artificial intelligence technology, voice-based intelligent systems (VISs), such as AI speakers and virtual assistants, are intervening in human life. VISs are emerging in a new way, called human–AI interaction, which is different from existing human–computer interaction. Using the Kansei engineering approach, we propose a method to evaluate user satisfaction during interaction between a VIS and a user-centered intelligent system. As a user satisfaction evaluation method, a VIS comprising four types of design parameters was developed. A total of 23 subjects were considered for interaction with the VIS, and user satisfaction was measured using Kansei words (KWs). The questionnaire scores collected through KWs were analyzed using exploratory factor analysis. ANOVA was used to analyze differences in emotion. On the “pleasurability” and “reliability” axes, it was confirmed that among the four design parameters, “sentence structure of the answer” and “number of trials to get the right answer for a question” affect the emotional satisfaction of users. Four satisfaction groups were derived according to the level of the design parameters. This study can be used as a reference for conducting an integrated emotional satisfaction assessment using emotional metrics such as biosignals and facial expressions.
Collapse
|
26
|
Zhang XY, Li JJ, Lu HT, Teng WJ, Liu SH. Positive effects of music therapist's selected auditory stimulation on the autonomic nervous system of patients with disorder of consciousness: a randomized controlled trial. Neural Regen Res 2021; 16:1266-1272. [PMID: 33318404 PMCID: PMC8284264 DOI: 10.4103/1673-5374.301021] [Citation(s) in RCA: 5] [Impact Index Per Article: 1.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/25/2023] Open
Abstract
The current randomized controlled trial was performed at the China Rehabilitation Science Institute, China to test the hypothesis that musical auditory stimulation has positive effects on the autonomic nervous system of patients with disorder of consciousness. Although past studies have recommended that patients with disorder of consciousness listen to patient-preferred music, this practice is not universally accepted by researchers. Twenty patients with severe disorder of consciousness listened to either therapist-selected (n = 10, 6 males and 4 females; 43.33 ± 18.76 years old) or patient-preferred (n = 10, 5 males and 5 females, 48.83 ± 18.79 years old) musical therapy, 30 minutes/day, 5 times/week for 6 weeks. The results showed no obvious differences in heart rate variability-related parameters including heart rate, standard deviation of normal-to-normal R-R intervals, and the root-mean-square of successive heartbeat interval differences of successive heartbeat intervals between the two groups of patients. However, percentage of differences exceeding 50 ms between adjacent normal number of intervals, low-frequency power/high-frequency power, high-frequency power norm, low-frequency power norm, and total power were higher in patients receiving therapist-selected music than in patients receiving their own preferred music. In contrast, this relationship was reversed for the high-frequency power and very-low-frequency band. These results suggest that compared with preferred musical stimulation, therapist-selected musical stimulation resulted in higher interactive activity of the autonomic nervous system. Therefore, therapist-selected musical stimulation should be used to arouse the autonomic nervous system of patients with disorder of consciousness. This study was approved by the Institutional Ethics Committee of China Rehabilitation Research Center, China (approval No. 2018-022-1) on March 12, 2018 and registered with the Chinese Clinical Trial Registry (registration number ChiCTR1800017809) on August 15, 2018.
Collapse
Affiliation(s)
- Xiao-Ying Zhang
- School of Rehabilitation Medicine, Capital Medical University; China Rehabilitation Science Institute; Beijing Key Laboratory of Neural Injury and Rehabilitation; Center of Neural Injury and Repair, Beijing Institute for Brain Disorders; Music Therapy Center, Department of Psychology, China Rehabilitation Research Center, Beijing, China
| | - Jian-Jun Li
- School of Rehabilitation Medicine, Capital Medical University; China Rehabilitation Science Institute; Beijing Key Laboratory of Neural Injury and Rehabilitation; Center of Neural Injury and Repair, Beijing Institute for Brain Disorders, Beijing, China
| | - Hai-Tao Lu
- School of Rehabilitation Medicine, Capital Medical University; Department of Neurorehabilitation, China Rehabilitation Research Center, Beijing, China
| | - Wen-Jia Teng
- School of Rehabilitation Medicine, Capital Medical University; Music Therapy Center, Department of Psychology, China Rehabilitation Research Center, Beijing, China
| | - Song-Huai Liu
- School of Rehabilitation Medicine, Capital Medical University; Music Therapy Center, Department of Psychology, China Rehabilitation Research Center, Beijing, China
| |
Collapse
|
27
|
EEG-Based Emotion Classification for Alzheimer's Disease Patients Using Conventional Machine Learning and Recurrent Neural Network Models. SENSORS 2020; 20:s20247212. [PMID: 33339334 PMCID: PMC7766766 DOI: 10.3390/s20247212] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 09/29/2020] [Revised: 12/08/2020] [Accepted: 12/12/2020] [Indexed: 11/16/2022]
Abstract
As the number of patients with Alzheimer's disease (AD) increases, the effort needed to care for these patients increases as well. At the same time, advances in information and sensor technologies have reduced caring costs, providing a potential pathway for developing healthcare services for AD patients. For instance, if a virtual reality (VR) system can provide emotion-adaptive content, the time that AD patients spend interacting with VR content is expected to be extended, allowing caregivers to focus on other tasks. As the first step towards this goal, in this study, we develop a classification model that detects AD patients' emotions (e.g., happy, peaceful, or bored). We first collected electroencephalography (EEG) data from 30 Korean female AD patients who watched emotion-evoking videos at a medical rehabilitation center. We applied conventional machine learning algorithms, such as a multilayer perceptron (MLP) and support vector machine, along with deep learning models of recurrent neural network (RNN) architectures. The best performance was obtained from MLP, which achieved an average accuracy of 70.97%; the RNN model's accuracy reached only 48.18%. Our study results open a new stream of research in the field of EEG-based emotion detection for patients with neurological disorders.
Collapse
|
28
|
Al-Nafjan A, Alharthi K, Kurdi H. Lightweight Building of an Electroencephalogram-Based Emotion Detection System. Brain Sci 2020; 10:brainsci10110781. [PMID: 33114646 PMCID: PMC7693518 DOI: 10.3390/brainsci10110781] [Citation(s) in RCA: 10] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/20/2020] [Revised: 10/23/2020] [Accepted: 10/23/2020] [Indexed: 11/24/2022] Open
Abstract
Brain–computer interface (BCI) technology provides a direct interface between the brain and an external device. BCIs have facilitated the monitoring of conscious brain electrical activity via electroencephalogram (EEG) signals and the detection of human emotion. Recently, great progress has been made in the development of novel paradigms for EEG-based emotion detection. These studies have also attempted to apply BCI research findings in varied contexts. Interestingly, advances in BCI technologies have increased the interest of scientists because such technologies’ practical applications in human–machine relationships seem promising. This emphasizes the need for a building process for an EEG-based emotion detection system that is lightweight, in terms of a smaller EEG dataset size and no involvement of feature extraction methods. In this study, we investigated the feasibility of using a spiking neural network to build an emotion detection system from a smaller version of the DEAP dataset with no involvement of feature extraction methods while maintaining decent accuracy. The results showed that by using a NeuCube-based spiking neural network, we could detect the valence emotion level using only 60 EEG samples with 84.62% accuracy, which is a comparable accuracy to that of previous studies.
Collapse
Affiliation(s)
- Abeer Al-Nafjan
- Computer Science Department, Imam Muhammad ibn Saud Islamic University, Riyadh 11432, Saudi Arabia;
| | - Khulud Alharthi
- Computer Science Department, King Saud University, Riyadh 11543, Saudi Arabia;
- Computer Science Department, Taif University, Taif 26571, Saudi Arabia
| | - Heba Kurdi
- Computer Science Department, King Saud University, Riyadh 11543, Saudi Arabia;
- Mechanical Engineering Department, Massachusetts Institute of Technology (MIT), Cambridge, MA 02139, USA
- Correspondence: or ; Tel.: +966-11-805-9637
| |
Collapse
|
29
|
Bai Y, Lin Y, Ziemann U. Managing disorders of consciousness: the role of electroencephalography. J Neurol 2020; 268:4033-4065. [PMID: 32915309 PMCID: PMC8505374 DOI: 10.1007/s00415-020-10095-z] [Citation(s) in RCA: 42] [Impact Index Per Article: 10.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/10/2020] [Revised: 07/18/2020] [Accepted: 07/18/2020] [Indexed: 02/07/2023]
Abstract
Disorders of consciousness (DOC) are an important but still underexplored entity in neurology. Novel electroencephalography (EEG) measures are currently being employed for improving diagnostic classification, estimating prognosis and supporting medicolegal decision-making in DOC patients. However, complex recording protocols, a confusing variety of EEG measures, and complicated analysis algorithms create roadblocks against broad application. We conducted a systematic review based on English-language studies in PubMed, Medline and Web of Science databases. The review structures the available knowledge based on EEG measures and analysis principles, and aims at promoting its translation into clinical management of DOC patients.
Collapse
Affiliation(s)
- Yang Bai
- International Vegetative State and Consciousness Science Institute, Hangzhou Normal University, Hangzhou, China
- Department of Neurology and Stroke, University of Tübingen, Hoppe‑Seyler‑Str. 3, 72076, Tübingen, Germany
- Hertie Institute for Clinical Brain Research, University of Tübingen, 72076, Tübingen, Germany
| | - Yajun Lin
- International Vegetative State and Consciousness Science Institute, Hangzhou Normal University, Hangzhou, China
| | - Ulf Ziemann
- Department of Neurology and Stroke, University of Tübingen, Hoppe‑Seyler‑Str. 3, 72076, Tübingen, Germany.
- Hertie Institute for Clinical Brain Research, University of Tübingen, 72076, Tübingen, Germany.
| |
Collapse
|
30
|
Leroy A, Cheron G. EEG dynamics and neural generators of psychological flow during one tightrope performance. Sci Rep 2020; 10:12449. [PMID: 32709919 PMCID: PMC7381607 DOI: 10.1038/s41598-020-69448-3] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.8] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/06/2020] [Accepted: 06/19/2020] [Indexed: 12/13/2022] Open
Abstract
Psychological “flow” emerges from a goal requiring action, and a match between skills and challenge. Using high-density electroencephalographic (EEG) recording, we quantified the neural generators characterizing psychological “flow” compared to a mindful “stress” state during a professional tightrope performance. Applying swLORETA based on self-reported mental states revealed the right superior temporal gyrus (BA38), right globus pallidus, and putamen as generators of delta, alpha, and beta oscillations, respectively, when comparing “flow” versus “stress”. Comparison of “stress” versus “flow” identified the middle temporal gyrus (BA39) as the delta generator, and the medial frontal gyrus (BA10) as the alpha and beta generator. These results support that “flow” emergence required transient hypo-frontality. Applying swLORETA on the motor command represented by the tibialis anterior EMG burst identified the ipsilateral cerebellum and contralateral sensorimotor cortex in association with on-line control exerted during both “flow” and “stress”, while the basal ganglia was identified only during “flow”.
Collapse
Affiliation(s)
- A Leroy
- Laboratory of Neurophysiology and Movement Biomechanics, Université Libre de Bruxelles, Brussels, Belgium.,Haute Ecole Provinciale du Hainaut-Condorcet, Mons, Belgium
| | - G Cheron
- Laboratory of Neurophysiology and Movement Biomechanics, Université Libre de Bruxelles, Brussels, Belgium. .,Laboratory of Electrophysiology, Université de Mons, Mons, Belgium.
| |
Collapse
|
31
|
Belkacem AN, Jamil N, Palmer JA, Ouhbi S, Chen C. Brain Computer Interfaces for Improving the Quality of Life of Older Adults and Elderly Patients. Front Neurosci 2020; 14:692. [PMID: 32694979 PMCID: PMC7339951 DOI: 10.3389/fnins.2020.00692] [Citation(s) in RCA: 34] [Impact Index Per Article: 8.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2020] [Accepted: 06/08/2020] [Indexed: 02/01/2023] Open
Abstract
All people experience aging, and the related physical and health changes, including changes in memory and brain function. These changes may become debilitating leading to an increase in dependence as people get older. Many external aids and tools have been developed to allow older adults and elderly patients to continue to live normal and comfortable lives. This mini-review describes some of the recent studies on cognitive decline and motor control impairment with the goal of advancing non-invasive brain computer interface (BCI) technologies to improve health and wellness of older adults and elderly patients. First, we describe the state of the art in cognitive prosthetics for psychiatric diseases. Then, we describe the state of the art of possible assistive BCI applications for controlling an exoskeleton, a wheelchair and smart home for elderly people with motor control impairments. The basic age-related brain and body changes, the effects of age on cognitive and motor abilities, and several BCI paradigms with typical tasks and outcomes are thoroughly described. We also discuss likely future trends and technologies to assist healthy older adults and elderly patients using innovative BCI applications with minimal technical oversight.
Collapse
Affiliation(s)
- Abdelkader Nasreddine Belkacem
- Department of Computer and Network Engineering, College of Information Technology, United Arab Emirates University, Al Ain, United Arab Emirates
| | - Nuraini Jamil
- Department of Computer Science and Software Engineering, College of Information Technology, United Arab Emirates University, Al Ain, United Arab Emirates
| | - Jason A. Palmer
- Department of Neurological Diagnosis and Restoration, Osaka University, Suita, Japan
| | - Sofia Ouhbi
- Department of Computer Science and Software Engineering, College of Information Technology, United Arab Emirates University, Al Ain, United Arab Emirates
| | - Chao Chen
- Key Laboratory of Complex System Control Theory and Application, Tianjin University of Technology, Tianjin, China
| |
Collapse
|
32
|
Murovec N, Heilinger A, Xu R, Ortner R, Spataro R, La Bella V, Miao Y, Jin J, Chatelle C, Laureys S, Allison BZ, Guger C. Effects of a Vibro-Tactile P300 Based Brain-Computer Interface on the Coma Recovery Scale-Revised in Patients With Disorders of Consciousness. Front Neurosci 2020; 14:294. [PMID: 32327970 PMCID: PMC7161577 DOI: 10.3389/fnins.2020.00294] [Citation(s) in RCA: 9] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/14/2019] [Accepted: 03/13/2020] [Indexed: 11/22/2022] Open
Abstract
Persons diagnosed with disorders of consciousness (DOC) typically suffer from motor and cognitive disabilities. Recent research has shown that non-invasive brain-computer interface (BCI) technology could help assess these patients' cognitive functions and command following abilities. 20 DOC patients participated in the study and performed 10 vibro-tactile P300 BCI sessions over 10 days with 8-12 runs each day. Vibrotactile tactors were placed on the each patient's left and right wrists and one foot. Patients were instructed, via earbuds, to concentrate and silently count vibrotactile pulses on either their left or right wrist that presented a target stimulus and to ignore the others. Changes of the BCI classification accuracy were investigated over the 10 days. In addition, the Coma Recovery Scale-Revised (CRS-R) score was measured before and after the 10 vibro-tactile P300 sessions. In the first run, 10 patients had a classification accuracy above chance level (>12.5%). In the best run, every patient reached an accuracy ≥60%. The grand average accuracy in the first session for all patients was 40%. In the best session, the grand average accuracy was 88% and the median accuracy across all sessions was 21%. The CRS-R scores compared before and after 10 VT3 sessions for all 20 patients, are showing significant improvement (p = 0.024). Twelve of the twenty patients showed an improvement of 1 to 7 points in the CRS-R score after the VT3 BCI sessions (mean: 2.6). Six patients did not show a change of the CRS-R and two patients showed a decline in the score by 1 point. Every patient achieved at least 60% accuracy at least once, which indicates successful command following. This shows the importance of repeated measures when DOC patients are assessed. The improvement of the CRS-R score after the 10 VT3 sessions is an important issue for future experiments to test the possible therapeutic applications of vibro-tactile and related BCIs with a larger patient group.
Collapse
Affiliation(s)
- Nensi Murovec
- g. tec Medical Engineering GmbH, Schiedlberg, Austria
- Guger Technologies OG, Graz, Austria
| | | | - Ren Xu
- Guger Technologies OG, Graz, Austria
| | - Rupert Ortner
- g. tec Medical Engineering Spain S.L., Barcelona, Spain
| | - Rossella Spataro
- g. tec Medical Engineering GmbH, Schiedlberg, Austria
- IRCCS Centro Neurolesi Bonino Pulejo, Palermo, Italy
| | - Vincenzo La Bella
- ALS Clinical Research Center, Bi.N.D., University of Palermo, Palermo, Italy
| | - Yangyang Miao
- Department of Automation, East China University of Science and Technology, Shanghai, China
| | - Jing Jin
- Department of Automation, East China University of Science and Technology, Shanghai, China
| | - Camille Chatelle
- GIGA Consciousness, Coma Science Group, University of Liège, Liège, Belgium
| | - Steven Laureys
- GIGA Consciousness, Coma Science Group, University of Liège, Liège, Belgium
- French Association of Locked-in Syndrome (ALIS), Paris, France
| | - Brendan Z. Allison
- Department of Cognitive Science, University of California, San Diego, La Jolla, CA, United States
| | - Christoph Guger
- g. tec Medical Engineering GmbH, Schiedlberg, Austria
- Guger Technologies OG, Graz, Austria
| |
Collapse
|
33
|
Pan J, Xie Q, Qin P, Chen Y, He Y, Huang H, Wang F, Ni X, Cichocki A, Yu R, Li Y. Prognosis for patients with cognitive motor dissociation identified by brain-computer interface. Brain 2020; 143:1177-1189. [PMID: 32101603 PMCID: PMC7174053 DOI: 10.1093/brain/awaa026] [Citation(s) in RCA: 71] [Impact Index Per Article: 17.8] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/04/2019] [Revised: 12/08/2019] [Accepted: 12/17/2019] [Indexed: 01/15/2023] Open
Abstract
Cognitive motor dissociation describes a subset of patients with disorders of consciousness who show neuroimaging evidence of consciousness but no detectable command-following behaviours. Although essential for family counselling, decision-making, and the design of rehabilitation programmes, the prognosis for patients with cognitive motor dissociation remains under-investigated. The current study included 78 patients with disorders of consciousness who showed no detectable command-following behaviours. These patients included 45 patients with unresponsive wakefulness syndrome and 33 patients in a minimally conscious state, as diagnosed using the Coma Recovery Scale-Revised. Each patient underwent an EEG-based brain-computer interface experiment, in which he or she was instructed to perform an item-selection task (i.e. select a photograph or a number from two candidates). Patients who achieved statistically significant brain-computer interface accuracies were identified as cognitive motor dissociation. Two evaluations using the Coma Recovery Scale-Revised, one before the experiment and the other 3 months later, were carried out to measure the patients' behavioural improvements. Among the 78 patients with disorders of consciousness, our results showed that within the unresponsive wakefulness syndrome patient group, 15 of 18 patients with cognitive motor dissociation (83.33%) regained consciousness, while only five of the other 27 unresponsive wakefulness syndrome patients without significant brain-computer interface accuracies (18.52%) regained consciousness. Furthermore, within the minimally conscious state patient group, 14 of 16 patients with cognitive motor dissociation (87.5%) showed improvements in their Coma Recovery Scale-Revised scores, whereas only four of the other 17 minimally conscious state patients without significant brain-computer interface accuracies (23.53%) had improved Coma Recovery Scale-Revised scores. Our results suggest that patients with cognitive motor dissociation have a better outcome than other patients. Our findings extend current knowledge of the prognosis for patients with cognitive motor dissociation and have important implications for brain-computer interface-based clinical diagnosis and prognosis for patients with disorders of consciousness.
Collapse
Affiliation(s)
- Jiahui Pan
- Center for Brain-Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China
- School of Software, South China Normal University, Guangzhou, China
| | - Qiuyou Xie
- Department of Rehabilitation Medicine, Zhujiang Hospital, Southern Medical University, Guangzhou, China
- Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, China
| | - Pengmin Qin
- Centre for Studies of Psychological Applications, Guangdong Key Laboratory of Mental Health and Cognitive Science, School of Psychology, South China Normal University, Guangzhou, China
| | - Yan Chen
- Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, China
| | - Yanbin He
- Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, China
- Department of Traumatic Brain Injury Rehabilitation and Severe Rehabilitation, Guangdong Work Injury Rehabilitation Hospital, Guangzhou, China
| | - Haiyun Huang
- Center for Brain-Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China
| | - Fei Wang
- Center for Brain-Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China
- School of Software, South China Normal University, Guangzhou, China
| | - Xiaoxiao Ni
- Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, China
| | - Andrzej Cichocki
- Skolkovo Institute of Science and Technology (Skoltech), Moscow 143026, Russia
- Nicolaus Copernicus University (UMK), Torun 87-100, Poland
| | - Ronghao Yu
- Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, China
| | - Yuanqing Li
- Center for Brain-Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, China
| |
Collapse
|
34
|
Sun Y, Ayaz H, Akansu AN. Multimodal Affective State Assessment Using fNIRS + EEG and Spontaneous Facial Expression. Brain Sci 2020; 10:E85. [PMID: 32041316 PMCID: PMC7071625 DOI: 10.3390/brainsci10020085] [Citation(s) in RCA: 14] [Impact Index Per Article: 3.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/05/2019] [Revised: 01/31/2020] [Accepted: 02/01/2020] [Indexed: 01/04/2023] Open
Abstract
Human facial expressions are regarded as a vital indicator of one's emotion and intention, and even reveal the state of health and wellbeing. Emotional states have been associated with information processing within and between subcortical and cortical areas of the brain, including the amygdala and prefrontal cortex. In this study, we evaluated the relationship between spontaneous human facial affective expressions and multi-modal brain activity measured via non-invasive and wearable sensors: functional near-infrared spectroscopy (fNIRS) and electroencephalography (EEG) signals. The affective states of twelve male participants detected via fNIRS, EEG, and spontaneous facial expressions were investigated in response to both image-content stimuli and video-content stimuli. We propose a method to jointly evaluate fNIRS and EEG signals for affective state detection (emotional valence as positive or negative). Experimental results reveal a strong correlation between spontaneous facial affective expressions and the perceived emotional valence. Moreover, the affective states were estimated by the fNIRS, EEG, and fNIRS + EEG brain activity measurements. We show that the proposed EEG + fNIRS hybrid method outperforms fNIRS-only and EEG-only approaches. Our findings indicate that the dynamic (video-content based) stimuli triggers a larger affective response than the static (image-content based) stimuli. These findings also suggest joint utilization of facial expression and wearable neuroimaging, fNIRS, and EEG, for improved emotional analysis and affective brain-computer interface applications.
Collapse
Affiliation(s)
- Yanjia Sun
- Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA;
| | - Hasan Ayaz
- School of Biomedical Engineering, Science and Health Systems, Drexel University, Philadelphia, PA 19104, USA;
- Department of Psychology, College of Arts and Sciences, Drexel University, Philadelphia, PA 19104, USA
- Department of Family and Community Health, University of Pennsylvania, Philadelphia, PA 19104, USA
- Center for Injury Research and Prevention, Children’s Hospital of Philadelphia, Philadelphia, PA 19104, USA
| | - Ali N. Akansu
- Department of Electrical and Computer Engineering, New Jersey Institute of Technology, Newark, NJ 07102, USA;
| |
Collapse
|
35
|
Annen J, Laureys S, Gosseries O. Brain-computer interfaces for consciousness assessment and communication in severely brain-injured patients. BRAIN-COMPUTER INTERFACES 2020; 168:137-152. [DOI: 10.1016/b978-0-444-63934-9.00011-1] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
36
|
Li Z, Zhang S, Pan J. Advances in Hybrid Brain-Computer Interfaces: Principles, Design, and Applications. COMPUTATIONAL INTELLIGENCE AND NEUROSCIENCE 2019; 2019:3807670. [PMID: 31687006 PMCID: PMC6800963 DOI: 10.1155/2019/3807670] [Citation(s) in RCA: 15] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/20/2019] [Revised: 09/09/2019] [Accepted: 09/17/2019] [Indexed: 11/23/2022]
Abstract
Conventional brain-computer interface (BCI) systems have been facing two fundamental challenges: the lack of high detection performance and the control command problem. To this end, the researchers have proposed a hybrid brain-computer interface (hBCI) to address these challenges. This paper mainly discusses the research progress of hBCI and reviews three types of hBCI, namely, hBCI based on multiple brain models, multisensory hBCI, and hBCI based on multimodal signals. By analyzing the general principles, paradigm designs, experimental results, advantages, and applications of the latest hBCI system, we found that using hBCI technology can improve the detection performance of BCI and achieve multidegree/multifunctional control, which is significantly superior to single-mode BCIs.
Collapse
Affiliation(s)
- Zina Li
- South China Normal University, Guangzhou 510631, China
| | - Shuqing Zhang
- South China Normal University, Guangzhou 510631, China
| | - Jiahui Pan
- South China Normal University, Guangzhou 510631, China
| |
Collapse
|
37
|
Combining Facial Expressions and Electroencephalography to Enhance Emotion Recognition. FUTURE INTERNET 2019. [DOI: 10.3390/fi11050105] [Citation(s) in RCA: 27] [Impact Index Per Article: 5.4] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/16/2022] Open
Abstract
Emotion recognition plays an essential role in human–computer interaction. Previous studies have investigated the use of facial expression and electroencephalogram (EEG) signals from single modal for emotion recognition separately, but few have paid attention to a fusion between them. In this paper, we adopted a multimodal emotion recognition framework by combining facial expression and EEG, based on a valence-arousal emotional model. For facial expression detection, we followed a transfer learning approach for multi-task convolutional neural network (CNN) architectures to detect the state of valence and arousal. For EEG detection, two learning targets (valence and arousal) were detected by different support vector machine (SVM) classifiers, separately. Finally, two decision-level fusion methods based on the enumerate weight rule or an adaptive boosting technique were used to combine facial expression and EEG. In the experiment, the subjects were instructed to watch clips designed to elicit an emotional response and then reported their emotional state. We used two emotion datasets—a Database for Emotion Analysis using Physiological Signals (DEAP) and MAHNOB-human computer interface (MAHNOB-HCI)—to evaluate our method. In addition, we also performed an online experiment to make our method more robust. We experimentally demonstrated that our method produces state-of-the-art results in terms of binary valence/arousal classification, based on DEAP and MAHNOB-HCI data sets. Besides this, for the online experiment, we achieved 69.75% accuracy for the valence space and 70.00% accuracy for the arousal space after fusion, each of which has surpassed the highest performing single modality (69.28% for the valence space and 64.00% for the arousal space). The results suggest that the combination of facial expressions and EEG information for emotion recognition compensates for their defects as single information sources. The novelty of this work is as follows. To begin with, we combined facial expression and EEG to improve the performance of emotion recognition. Furthermore, we used transfer learning techniques to tackle the problem of lacking data and achieve higher accuracy for facial expression. Finally, in addition to implementing the widely used fusion method based on enumerating different weights between two models, we also explored a novel fusion method, applying boosting technique.
Collapse
|
38
|
Xie Q, Pan J, Chen Y, He Y, Ni X, Zhang J, Wang F, Li Y, Yu R. A gaze-independent audiovisual brain-computer Interface for detecting awareness of patients with disorders of consciousness. BMC Neurol 2018; 18:144. [PMID: 30296948 PMCID: PMC6176505 DOI: 10.1186/s12883-018-1144-y] [Citation(s) in RCA: 19] [Impact Index Per Article: 3.2] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2017] [Accepted: 08/29/2018] [Indexed: 11/10/2022] Open
Abstract
BACKGROUND Currently, it is challenging to detect the awareness of patients who suffer disorders of consciousness (DOC). Brain-computer interfaces (BCIs), which do not depend on the behavioral response of patients, may serve for detecting the awareness in patients with DOC. However, we must develop effective BCIs for these patients because their ability to use BCIs does not as good as healthy users. METHODS Because patients with DOC generally do not exhibit eye movements, a gaze-independent audiovisual BCI is put forward in the study where semantically congruent and incongruent audiovisual number stimuli were sequentially presented to evoke event-related potentials (ERPs). Subjects were required to pay attention to congruent audiovisual stimuli (target) and ignore the incongruent audiovisual stimuli (non-target). The BCI system was evaluated by analyzing online and offline data from 10 healthy subjects followed by being applied to online awareness detection in 8 patients with DOC. RESULTS According to the results on healthy subjects, the audiovisual BCI system outperformed the corresponding auditory-only and visual-only systems. Multiple ERP components, including the P300, N400 and late positive complex (LPC), were observed using the audiovisual system, strengthening different brain responses to target stimuli and non-target stimuli. The results revealed the abilities of three of eight patients to follow commands and recognize numbers. CONCLUSIONS This gaze-independent audiovisual BCI system represents a useful auxiliary bedside tool to detect the awareness of patients with DOC.
Collapse
Affiliation(s)
- Qiuyou Xie
- Coma Research Group, Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, 510010 China
| | - Jiahui Pan
- School of Software, South China Normal University, Guangzhou, 510641 China
- Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, 510640 China
| | - Yan Chen
- Coma Research Group, Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, 510010 China
| | - Yanbin He
- Coma Research Group, Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, 510010 China
| | - Xiaoxiao Ni
- Coma Research Group, Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, 510010 China
| | - Jiechun Zhang
- Coma Research Group, Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, 510010 China
| | - Fei Wang
- Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, 510640 China
| | - Yuanqing Li
- Center for Brain Computer Interfaces and Brain Information Processing, South China University of Technology, Guangzhou, 510640 China
| | - Ronghao Yu
- Coma Research Group, Centre for Hyperbaric Oxygen and Neurorehabilitation, Guangzhou General Hospital of Guangzhou Military Command, Guangzhou, 510010 China
| |
Collapse
|