1
|
Dar MN, Akram MU, Subhani AR, Khawaja SG, Reyes-Aldasoro CC, Gul S. Insights from EEG analysis of evoked memory recalls using deep learning for emotion charting. Sci Rep 2024; 14:17080. [PMID: 39048599 PMCID: PMC11269615 DOI: 10.1038/s41598-024-61832-7] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Accepted: 05/10/2024] [Indexed: 07/27/2024] Open
Abstract
Affect recognition in a real-world, less constrained environment is the principal prerequisite of the industrial-level usefulness of this technology. Monitoring the psychological profile using smart, wearable electroencephalogram (EEG) sensors during daily activities without external stimuli, such as memory-induced emotions, is a challenging research gap in emotion recognition. This paper proposed a deep learning framework for improved memory-induced emotion recognition leveraging a combination of 1D-CNN and LSTM as feature extractors integrated with an Extreme Learning Machine (ELM) classifier. The proposed deep learning architecture, combined with the EEG preprocessing, such as the removal of the average baseline signal from each sample and extraction of EEG rhythms (delta, theta, alpha, beta, and gamma), aims to capture repetitive and continuous patterns for memory-induced emotion recognition, underexplored with deep learning techniques. This work has analyzed EEG signals using a wearable, ultra-mobile sports cap while recalling autobiographical emotional memories evoked by affect-denoting words, with self-annotation on the scale of valence and arousal. With extensive experimentation using the same dataset, the proposed framework empirically outperforms existing techniques for the emerging area of memory-induced emotion recognition with an accuracy of 65.6%. The EEG rhythms analysis, such as delta, theta, alpha, beta, and gamma, achieved 65.5%, 52.1%, 65.1%, 64.6%, and 65.0% accuracies for classification with four quadrants of valence and arousal. These results underscore the significant advancement achieved by our proposed method for the real-world environment of memory-induced emotion recognition.
Collapse
Affiliation(s)
- Muhammad Najam Dar
- National University of Sciences and Technology, Islamabad, 44000, Pakistan.
| | | | - Ahmad Rauf Subhani
- National University of Sciences and Technology, Islamabad, 44000, Pakistan
| | - Sajid Gul Khawaja
- National University of Sciences and Technology, Islamabad, 44000, Pakistan
| | - Constantino Carlos Reyes-Aldasoro
- Department of Computer Science, School of Science and Technology, City, University of London, Northampton Square, London, EC1V 0HB, UK
| | - Sarah Gul
- Department of Biological Sciences, FBAS, International Islamic University, Islamabad, Pakistan
| |
Collapse
|
2
|
Yu Z, Kachenoura A, Jeannès RLB, Shu H, Berraute P, Nica A, Merlet I, Albera L, Karfoul A. Electrophysiological brain imaging based on simulation-driven deep learning in the context of epilepsy. Neuroimage 2024; 285:120490. [PMID: 38103624 DOI: 10.1016/j.neuroimage.2023.120490] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/07/2023] [Revised: 11/30/2023] [Accepted: 12/06/2023] [Indexed: 12/19/2023] Open
Abstract
Identifying the location, the spatial extent and the electrical activity of distributed brain sources in the context of epilepsy through ElectroEncephaloGraphy (EEG) recordings is a challenging task because of the highly ill-posed nature of the underlying Electrophysiological Source Imaging (ESI) problem. To guarantee a unique solution, most existing ESI methods pay more attention to solve this inverse problem by imposing physiological constraints. This paper proposes an efficient ESI approach based on simulation-driven deep learning. Epileptic High-resolution 256-channels scalp EEG (Hr-EEG) signals are simulated in a realistic manner to train the proposed patient-specific model. More particularly, a computational neural mass model developed in our team is used to generate the temporal dynamics of the activity of each dipole while the forward problem is solved using a patient-specific three-shell realistic head model and the boundary element method. A Temporal Convolutional Network (TCN) is considered in the proposed model to capture local spatial patterns. To enable the model to observe the EEG signals from different scale levels, the multi-scale strategy is leveraged to capture the overall features and fine-grain features by adjusting the convolutional kernel size. Then, the Long Short-Term Memory (LSTM) is used to extract temporal dependencies among the computed spatial features. The performance of the proposed method is evaluated through three different scenarios of realistic synthetic interictal Hr-EEG data as well as on real interictal Hr-EEG data acquired in three patients with drug-resistant partial epilepsy, during their presurgical evaluation. A performance comparison study is also conducted with two other deep learning-based methods and four classical ESI techniques. The proposed model achieved a Dipole Localization Error (DLE) of 1.39 and Normalized Hamming Distance (NHD) of 0.28 in the case of one patch with SNR of 10 dB. In the case of two uncorrelated patches with an SNR of 10 dB, obtained DLE and NHD were respectively 1.50 and 0.28. Even in the more challenging scenario of two correlated patches with an SNR of 10 dB, the proposed approach still achieved a DLE of 3.74 and an NHD of 0.43. The results obtained on simulated data demonstrate that the proposed method outperforms the existing methods for different signal-to-noise and source configurations. The good behavior of the proposed method is also confirmed on real interictal EEG data. The robustness with respect to noise makes it a promising and alternative tool to localize epileptic brain areas and to reconstruct their electrical activities from EEG signals.
Collapse
Affiliation(s)
- Zuyi Yu
- Laboratory of Image Science and Technology, Southeast University, Nanjing 210096, PR China; Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing, Nanjing 210096, PR China; University Rennes, INSERM, LTSI-UMR 1099, Rennes F-35042, France; Centre de Recherche en Information Biomédicale Sino-français (CRIBs), Rennes F-35042, France
| | - Amar Kachenoura
- University Rennes, INSERM, LTSI-UMR 1099, Rennes F-35042, France; Centre de Recherche en Information Biomédicale Sino-français (CRIBs), Rennes F-35042, France
| | - Régine Le Bouquin Jeannès
- University Rennes, INSERM, LTSI-UMR 1099, Rennes F-35042, France; Centre de Recherche en Information Biomédicale Sino-français (CRIBs), Rennes F-35042, France
| | - Huazhong Shu
- Laboratory of Image Science and Technology, Southeast University, Nanjing 210096, PR China; Jiangsu Provincial Joint International Research Laboratory of Medical Information Processing, Nanjing 210096, PR China.
| | | | - Anca Nica
- University Rennes, INSERM, LTSI-UMR 1099, Rennes F-35042, France; Centre Hospitalier Universitaire (CHU) de Rennes, service de neurologie, pôle des neurosciences de Rennes, Rennes F-35042, France
| | - Isabelle Merlet
- University Rennes, INSERM, LTSI-UMR 1099, Rennes F-35042, France; Centre de Recherche en Information Biomédicale Sino-français (CRIBs), Rennes F-35042, France
| | - Laurent Albera
- University Rennes, INSERM, LTSI-UMR 1099, Rennes F-35042, France; Centre de Recherche en Information Biomédicale Sino-français (CRIBs), Rennes F-35042, France.
| | - Ahmad Karfoul
- University Rennes, INSERM, LTSI-UMR 1099, Rennes F-35042, France; Centre de Recherche en Information Biomédicale Sino-français (CRIBs), Rennes F-35042, France
| |
Collapse
|
3
|
Jiao M, Wan G, Guo Y, Wang D, Liu H, Xiang J, Liu F. A Graph Fourier Transform Based Bidirectional Long Short-Term Memory Neural Network for Electrophysiological Source Imaging. Front Neurosci 2022; 16:867466. [PMID: 35495022 PMCID: PMC9043242 DOI: 10.3389/fnins.2022.867466] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/01/2022] [Accepted: 03/14/2022] [Indexed: 11/13/2022] Open
Abstract
Electrophysiological source imaging (ESI) refers to the process of reconstructing underlying activated sources on the cortex given the brain signal measured by Electroencephalography (EEG) or Magnetoencephalography (MEG). Due to the ill-posed nature of ESI, solving ESI requires the design of neurophysiologically plausible regularization or priors to guarantee a unique solution. Recovering focally extended sources is more challenging, and traditionally uses a total variation regularization to promote spatial continuity of the activated sources. In this paper, we propose to use graph Fourier transform (GFT) based bidirectional long-short term memory (BiLSTM) neural network to solve the ESI problem. The GFT delineates the 3D source space into spatially high, medium and low frequency subspaces spanned by corresponding eigenvectors. The low frequency components can naturally serve as a spatially low-band pass filter to reconstruct extended areas of source activation. The BiLSTM is adopted to learn the mapping relationship between the projection of low-frequency graph space and the recorded EEG. Numerical results show the proposed GFT-BiLSTM outperforms other benchmark algorithms in synthetic data under varied signal-to-noise ratios (SNRs). Real data experiments also demonstrate its capability of localizing the epileptogenic zone of epilepsy patients with good accuracy.
Collapse
Affiliation(s)
- Meng Jiao
- School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, United States
- College of Electrical Engineering, Qingdao University, Qingdao, China
| | - Guihong Wan
- Department of Dermatology, Massachusetts General Hospital, Harvard Medical School, Boston, MA, United States
- Department of Biomedical Informatics, Harvard Medical School, Boston, MA, United States
| | - Yaxin Guo
- School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, United States
| | - Dongqing Wang
- College of Electrical Engineering, Qingdao University, Qingdao, China
| | - Hang Liu
- Department of Electrical and Computer Engineering, Stevens Institute of Technology, Hoboken, NJ, United States
| | - Jing Xiang
- MEG Center, Division of Neurology, Cincinnati Children’s Hospital Medical Center, Cincinnati, OH, United States
| | - Feng Liu
- School of Systems and Enterprises, Stevens Institute of Technology, Hoboken, NJ, United States
- *Correspondence: Feng Liu,
| |
Collapse
|
4
|
Yeung MK, Chu VW. Viewing neurovascular coupling through the lens of combined EEG-fNIRS: A systematic review of current methods. Psychophysiology 2022; 59:e14054. [PMID: 35357703 DOI: 10.1111/psyp.14054] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/15/2021] [Revised: 02/01/2022] [Accepted: 03/08/2022] [Indexed: 12/25/2022]
Abstract
Neurovascular coupling is a key physiological mechanism that occurs in the healthy human brain, and understanding this process has implications for understanding the aging and neuropsychiatric populations. Combined electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) has emerged as a promising, noninvasive tool for probing neurovascular interactions in humans. However, the utility of this approach critically depends on the methodological quality used for multimodal integration. Despite a growing number of combined EEG-fNIRS applications reported in recent years, the methodological rigor of past studies remains unclear, limiting the accurate interpretation of reported findings and hindering the translational application of this multimodal approach. To fill this knowledge gap, we critically evaluated various methodological aspects of previous combined EEG-fNIRS studies performed in healthy individuals. A literature search was conducted using PubMed and PsycINFO on June 28, 2021. Studies involving concurrent EEG and fNIRS measurements in awake and healthy individuals were selected. After screening and eligibility assessment, 96 studies were included in the methodological evaluation. Specifically, we critically reviewed various aspects of participant sampling, experimental design, signal acquisition, data preprocessing, outcome selection, data analysis, and results presentation reported in these studies. Altogether, we identified several notable strengths and limitations of the existing EEG-fNIRS literature. In light of these limitations and the features of combined EEG-fNIRS, recommendations are made to improve and standardize research practices to facilitate the use of combined EEG-fNIRS when studying healthy neurovascular coupling processes and alterations in neurovascular coupling among various populations.
Collapse
Affiliation(s)
- Michael K Yeung
- Department of Rehabilitation Sciences, The Hong Kong Polytechnic University, Hong Kong, China
| | - Vivian W Chu
- Department of Rehabilitation Sciences, The Hong Kong Polytechnic University, Hong Kong, China
| |
Collapse
|
5
|
de With LA, Thammasan N, Poel M. Detecting Fear of Heights Response to a Virtual Reality Environment Using Functional Near-Infrared Spectroscopy. FRONTIERS IN COMPUTER SCIENCE 2022. [DOI: 10.3389/fcomp.2021.652550] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
To enable virtual reality exposure therapy (VRET) that treats anxiety disorders by gradually exposing the patient to fear using virtual reality (VR), it is important to monitor the patient's fear levels during the exposure. Despite the evidence of a fear circuit in the brain as reflected by functional near-infrared spectroscopy (fNIRS), the measurement of fear response in highly immersive VR using fNIRS is limited, especially in combination with a head-mounted display (HMD). In particular, it is unclear to what extent fNIRS can differentiate users with and without anxiety disorders and detect fear response in a highly ecological setting using an HMD. In this study, we investigated fNIRS signals captured from participants with and without a fear of height response. To examine the extent to which fNIRS signals of both groups differ, we conducted an experiment during which participants with moderate fear of heights and participants without it were exposed to VR scenarios involving heights and no heights. The between-group statistical analysis shows that the fNIRS data of the control group and the experimental group are significantly different only in the channel located close to right frontotemporal lobe, where the grand average oxygenated hemoglobin Δ[HbO] contrast signal of the experimental group exceeds that of the control group. The within-group statistical analysis shows significant differences between the grand average Δ[HbO] contrast values during fear responses and those during no-fear responses, where the Δ[HbO] contrast values of the fear responses were significantly higher than those of the no-fear responses in the channels located towards the frontal part of the prefrontal cortex. Also, the channel located close to frontocentral lobe was found to show significant difference for the grand average deoxygenated hemoglobin contrast signals. Support vector machine-based classifier could detect fear responses at an accuracy up to 70% and 74% in subject-dependent and subject-independent classifications, respectively. The results demonstrate that cortical hemodynamic responses of a control group and an experimental group are different to a considerable extent, exhibiting the feasibility and ecological validity of the combination of VR-HMD and fNIRS to elicit and detect fear responses. This research thus paves a way toward the a brain-computer interface to effectively manipulate and control VRET.
Collapse
|
6
|
Mei J, Muller E, Ramaswamy S. Informing deep neural networks by multiscale principles of neuromodulatory systems. Trends Neurosci 2022; 45:237-250. [DOI: 10.1016/j.tins.2021.12.008] [Citation(s) in RCA: 5] [Impact Index Per Article: 2.5] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/09/2021] [Revised: 12/04/2021] [Accepted: 12/21/2021] [Indexed: 01/19/2023]
|
7
|
Kahya M, Lyons KE, Pahwa R, Akinwuntan AE, He J, Devos H. Pupillary Response to Postural Demand in Parkinson's Disease. Front Bioeng Biotechnol 2021; 9:617028. [PMID: 33987171 PMCID: PMC8111006 DOI: 10.3389/fbioe.2021.617028] [Citation(s) in RCA: 7] [Impact Index Per Article: 2.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/13/2020] [Accepted: 03/09/2021] [Indexed: 12/11/2022] Open
Abstract
Background: Individuals with Parkinson’s disease (PD) may need to spend more mental and physical effort (i.e., cognitive workload) to maintain postural control. Pupillary response reflects cognitive workload during postural control tasks in healthy controls but has not been investigated as a measure of postural demand in PD. Objectives: To compare pupillary response during increased postural demand using vision occlusion and dual tasking between individuals with PD and healthy controls. Methods: Thirty-three individuals with PD and thirty-five healthy controls were recruited. The four conditions lasted 60 s and involved single balance task with eyes open; single balance task with eyes occluded; dual task with eyes open; dual task with eyes occluded. The dual task comprised the Auditory Stroop test. Pupillary response was recorded using an eye tracker. The balance was assessed by using a force plate. Two-way Repeated Measures ANOVA and LSD post-hoc tests were employed to compare pupillary response and Center of Pressure (CoP) displacement across the four conditions and between individuals with PD and healthy controls. Results: Pupillary response was higher in individuals with PD compared to healthy controls (p = 0.009) and increased with more challenging postural conditions in both groups (p < 0.001). The post-hoc analysis demonstrated increased pupillary response in the single balance eyes occluded (p < 0.001), dual task eyes open (p = 0.01), and dual task eyes occluded (p < 0.001) conditions compared to single task eyes open condition. Conclusion: Overall, the PD group had increased pupillary response with increased postural demand compared to the healthy controls. In the future, pupillary response can be a potential tool to understand the neurophysiological underpinnings of falls risk in the PD population.
Collapse
Affiliation(s)
- Melike Kahya
- Hinda and Arthur Marcus Institute for Aging Research, Harvard Medical School, Boston, MA, United States
| | - Kelly E Lyons
- Department of Neurology, School of Medicine, University of Kansas Medical Center, Kansas City, KS, United States
| | - Rajesh Pahwa
- Department of Neurology, School of Medicine, University of Kansas Medical Center, Kansas City, KS, United States
| | - Abiodun E Akinwuntan
- Office of the Dean, School of Health Professions, University of Kansas Medical Center, Kansas City, KS, United States.,Department of Physical Therapy and Rehabilitation Science, School of Health Professions, University of Kansas Medical Center, Kansas City, KS, United States
| | - Jianghua He
- Department of Biostatistics and Data Science, University of Kansas Medical Center, Kansas City, KS, United States
| | - Hannes Devos
- Department of Physical Therapy and Rehabilitation Science, School of Health Professions, University of Kansas Medical Center, Kansas City, KS, United States
| |
Collapse
|