1
|
Gramann K, Lotte F, Dehais F, Ayaz H, Vukelić M, Karwowski W, Fairclough S, Brouwer AM, Roy RN. Editorial: Open science to support replicability in neuroergonomic research. FRONTIERS IN NEUROERGONOMICS 2024; 5:1459204. [PMID: 39139473 PMCID: PMC11319283 DOI: 10.3389/fnrgo.2024.1459204] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/03/2024] [Accepted: 07/08/2024] [Indexed: 08/15/2024]
Affiliation(s)
- Klaus Gramann
- Biological Psychology and Neuroergonomics, Technische Universitaet Berlin, Berlin, Germany
| | - Fabien Lotte
- Inria Center at the University of Bordeaux/LaBRI, Bordeaux, France
| | - Frederic Dehais
- Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, France
| | - Hasan Ayaz
- School of Biomedical Engineering Science and Health Systems, Drexel University, Philadelphia, PA, United States
| | - Mathias Vukelić
- Fraunhofer Institute for Industrial Engineering IAO, Stuttgart, Germany
| | - Waldemar Karwowski
- Department of Industrial Engineering and Management Systems, University of Central Florida, Orlando, FL, United States
| | - Stephen Fairclough
- School of Psychology, Liverpool John Moores University, Liverpool, United Kingdom
| | | | - Raphaëlle N. Roy
- Fédération ENAC ISAE-SUPAERO ONERA, Université de Toulouse, Toulouse, France
| |
Collapse
|
2
|
Demirezen G, Taşkaya Temizel T, Brouwer AM. Reproducible machine learning research in mental workload classification using EEG. FRONTIERS IN NEUROERGONOMICS 2024; 5:1346794. [PMID: 38660590 PMCID: PMC11039816 DOI: 10.3389/fnrgo.2024.1346794] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Accepted: 03/22/2024] [Indexed: 04/26/2024]
Abstract
This study addresses concerns about reproducibility in scientific research, focusing on the use of electroencephalography (EEG) and machine learning to estimate mental workload. We established guidelines for reproducible machine learning research using EEG and used these to assess the current state of reproducibility in mental workload modeling. We first started by summarizing the current state of reproducibility efforts in machine learning and in EEG. Next, we performed a systematic literature review on Scopus, Web of Science, ACM Digital Library, and Pubmed databases to find studies about reproducibility in mental workload prediction using EEG. All of this previous work was used to formulate guidelines, which we structured along the widely recognized Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. By using these guidelines, researchers can ensure transparency and comprehensiveness of their methodologies, therewith enhancing collaboration and knowledge-sharing within the scientific community, and enhancing the reliability, usability and significance of EEG and machine learning techniques in general. A second systematic literature review extracted machine learning studies that used EEG to estimate mental workload. We evaluated the reproducibility status of these studies using our guidelines. We highlight areas studied and overlooked and identify current challenges for reproducibility. Our main findings include limitations on reporting performance on unseen test data, open sharing of data and code, and reporting of resources essential for training and inference processes.
Collapse
Affiliation(s)
- Güliz Demirezen
- Department of Information Systems, Graduate School of Informatics, Middle East Technical University, Ankara, Türkiye
| | - Tuğba Taşkaya Temizel
- Department of Data Informatics, Graduate School of Informatics, Middle East Technical University, Ankara, Türkiye
| | - Anne-Marie Brouwer
- Human Performance, Netherlands Organisation for Applied Scientific Research (TNO), Soesterberg, Netherlands
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| |
Collapse
|
3
|
Ladouce S, Pietzker M, Manzey D, Dehais F. Evaluation of a headphones-fitted EEG system for the recording of auditory evoked potentials and mental workload assessment. Behav Brain Res 2024; 460:114827. [PMID: 38128886 DOI: 10.1016/j.bbr.2023.114827] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/13/2023] [Revised: 11/23/2023] [Accepted: 12/18/2023] [Indexed: 12/23/2023]
Abstract
Advancements in portable neuroimaging technologies open up new opportunities to gain insight into the neural dynamics and cognitive processes underlying day-to-day behaviors. In this study, we evaluated the relevance of a headphone- mounted electroencephalogram (EEG) system for monitoring mental workload. The participants (N = 12) were instructed to pay attention to auditory alarms presented sporadically while performing the Multi-Attribute Task Battery (MATB) whose difficulty was staged across three conditions to manipulate mental workload. The P300 Event-Related Potentials (ERP) elicited by the presentation of auditory alarms were used as probes of attentional resources available. The amplitude and latency of P300 ERPs were compared across experimental conditions. Our findings indicate that the P300 ERP component can be captured using a headphone-mounted EEG system. Moreover, neural responses to alarm could be used to classify mental workload with high accuracy (over 80%) at a single-trial level. Our analyses indicated that the signal-to-noise ratio acquired by the sponge-based sensors remained stable throughout the recordings. These results highlight the potential of portable neuroimaging technology for the development of neuroassistive applications while underscoring the current limitations and challenges associated with the integration of EEG sensors in everyday-life wearable technologies. Overall, our study contributes to the growing body of research exploring the feasibility and validity of wearable neuroimaging technologies for the study of human cognition and behavior in real-world settings.
Collapse
Affiliation(s)
- Simon Ladouce
- Human Factors and Neuroergonomics, ISAE-SUPAERO, 10 Av. Edouard Belin, Toulouse 31400, Haute-Garonne, France.
| | - Max Pietzker
- Department of Psychology and Ergonomics, Technical University Berlin, Strafte des 17.Juni 135, 10623 Berlin, Berlin, 10623 Berlin, Germany
| | - Dietrich Manzey
- Department of Psychology and Ergonomics, Technical University Berlin, Strafte des 17.Juni 135, 10623 Berlin, Berlin, 10623 Berlin, Germany
| | - Frederic Dehais
- Human Factors and Neuroergonomics, ISAE-SUPAERO, 10 Av. Edouard Belin, Toulouse 31400, Haute-Garonne, France; School of Biomedical Engineering, Science Health Systems, Drexel University, 3141 Chestnut St, Philadelphia 19104, PA, United States
| |
Collapse
|
4
|
Welter M, Lotte F. Ecological decoding of visual aesthetic preference with oscillatory electroencephalogram features-A mini-review. FRONTIERS IN NEUROERGONOMICS 2024; 5:1341790. [PMID: 38450005 PMCID: PMC10914990 DOI: 10.3389/fnrgo.2024.1341790] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/20/2023] [Accepted: 01/19/2024] [Indexed: 03/08/2024]
Abstract
In today's digital information age, human exposure to visual artifacts has reached an unprecedented quasi-omnipresence. Some of these cultural artifacts are elevated to the status of artworks which indicates a special appreciation of these objects. For many persons, the perception of such artworks coincides with aesthetic experiences (AE) that can positively affect health and wellbeing. AEs are composed of complex cognitive and affective mental and physiological states. More profound scientific understanding of the neural dynamics behind AEs would allow the development of passive Brain-Computer-Interfaces (BCI) that offer personalized art presentation to improve AE without the necessity of explicit user feedback. However, previous empirical research in visual neuroaesthetics predominantly investigated functional Magnetic Resonance Imaging and Event-Related-Potentials correlates of AE in unnaturalistic laboratory conditions which might not be the best features for practical neuroaesthetic BCIs. Furthermore, AE has, until recently, largely been framed as the experience of beauty or pleasantness. Yet, these concepts do not encompass all types of AE. Thus, the scope of these concepts is too narrow to allow personalized and optimal art experience across individuals and cultures. This narrative mini-review summarizes the state-of-the-art in oscillatory Electroencephalography (EEG) based visual neuroaesthetics and paints a road map toward the development of ecologically valid neuroaesthetic passive BCI systems that could optimize AEs, as well as their beneficial consequences. We detail reported oscillatory EEG correlates of AEs, as well as machine learning approaches to classify AE. We also highlight current limitations in neuroaesthetics and suggest future directions to improve EEG decoding of AE.
Collapse
Affiliation(s)
- Marc Welter
- Inria Center at the University of Bordeaux/LaBRI, Talence, France
| | | |
Collapse
|
5
|
Yamamoto MS, Sadatnejad K, Tanaka T, Islam MR, Dehais F, Tanaka Y, Lotte F. Modeling Complex EEG Data Distribution on the Riemannian Manifold Toward Outlier Detection and Multimodal Classification. IEEE Trans Biomed Eng 2024; 71:377-387. [PMID: 37450357 DOI: 10.1109/tbme.2023.3295769] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 07/18/2023]
Abstract
OBJECTIVE The usage of Riemannian geometry for Brain-computer interfaces (BCIs) has gained momentum in recent years. Most of the machine learning techniques proposed for Riemannian BCIs consider the data distribution on a manifold to be unimodal. However, the distribution is likely to be multimodal rather than unimodal since high-data variability is a crucial limitation of electroencephalography (EEG). In this paper, we propose a novel data modeling method for considering complex data distributions on a Riemannian manifold of EEG covariance matrices, aiming to improve BCI reliability. METHODS Our method, Riemannian spectral clustering (RiSC), represents EEG covariance matrix distribution on a manifold using a graph with proposed similarity measurement based on geodesic distances, then clusters the graph nodes through spectral clustering. This allows flexibility to model both a unimodal and a multimodal distribution on a manifold. RiSC can be used as a basis to design an outlier detector named outlier detection Riemannian spectral clustering (odenRiSC) and a multimodal classifier named multimodal classifier Riemannian spectral clustering (mcRiSC). All required parameters of odenRiSC/mcRiSC are selected in data-driven manner. Moreover, there is no need to pre-set a threshold for outlier detection and the number of modes for multimodal classification. RESULTS The experimental evaluation revealed odenRiSC can detect EEG outliers more accurately than existing methods and mcRiSC outperformed the standard unimodal classifier, especially on high-variability datasets. CONCLUSION odenRiSC/mcRiSC are anticipated to contribute to making real-life BCIs outside labs and neuroergonomics applications more robust. SIGNIFICANCE RiSC can work as a robust EEG outlier detector and multimodal classifier.
Collapse
|
6
|
Yu S, Wang Z, Wang F, Chen K, Yao D, Xu P, Zhang Y, Wang H, Zhang T. Multiclass classification of motor imagery tasks based on multi-branch convolutional neural network and temporal convolutional network model. Cereb Cortex 2024; 34:bhad511. [PMID: 38183186 DOI: 10.1093/cercor/bhad511] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/05/2023] [Revised: 12/06/2023] [Accepted: 12/08/2023] [Indexed: 01/07/2024] Open
Abstract
Motor imagery (MI) is a cognitive process wherein an individual mentally rehearses a specific movement without physically executing it. Recently, MI-based brain-computer interface (BCI) has attracted widespread attention. However, accurate decoding of MI and understanding of neural mechanisms still face huge challenges. These seriously hinder the clinical application and development of BCI systems based on MI. Thus, it is very necessary to develop new methods to decode MI tasks. In this work, we propose a multi-branch convolutional neural network (MBCNN) with a temporal convolutional network (TCN), an end-to-end deep learning framework to decode multi-class MI tasks. We first used MBCNN to capture the MI electroencephalography signals information on temporal and spectral domains through different convolutional kernels. Then, we introduce TCN to extract more discriminative features. The within-subject cross-session strategy is used to validate the classification performance on the dataset of BCI Competition IV-2a. The results showed that we achieved 75.08% average accuracy for 4-class MI task classification, outperforming several state-of-the-art approaches. The proposed MBCNN-TCN-Net framework successfully captures discriminative features and decodes MI tasks effectively, improving the performance of MI-BCIs. Our findings could provide significant potential for improving the clinical application and development of MI-based BCI systems.
Collapse
Affiliation(s)
- Shiqi Yu
- Microecology Research Center, Baiyun Branch, Nanfang Hospital, Southern Medical University, Guangzhou 510515, China
- Mental Health Education Center and School of Science, Xihua University, Chengdu 610039, China
| | - Zedong Wang
- Microecology Research Center, Baiyun Branch, Nanfang Hospital, Southern Medical University, Guangzhou 510515, China
| | - Fei Wang
- School of Computer and Software, Chengdu Jincheng College, Chengdu 610097, China
| | - Kai Chen
- Mental Health Education Center and School of Science, Xihua University, Chengdu 610039, China
| | - Dezhong Yao
- Key Laboratory for Neuroinformation of Ministry of Education, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Peng Xu
- Key Laboratory for Neuroinformation of Ministry of Education, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, China
| | - Yong Zhang
- Microecology Research Center, Baiyun Branch, Nanfang Hospital, Southern Medical University, Guangzhou 510515, China
| | - Hesong Wang
- Microecology Research Center, Baiyun Branch, Nanfang Hospital, Southern Medical University, Guangzhou 510515, China
| | - Tao Zhang
- Microecology Research Center, Baiyun Branch, Nanfang Hospital, Southern Medical University, Guangzhou 510515, China
- Mental Health Education Center and School of Science, Xihua University, Chengdu 610039, China
- Key Laboratory for Neuroinformation of Ministry of Education, School of Life Science and Technology, University of Electronic Science and Technology of China, Chengdu 611731, China
| |
Collapse
|
7
|
Vukelić M, Bui M, Vorreuther A, Lingelbach K. Combining brain-computer interfaces with deep reinforcement learning for robot training: a feasibility study in a simulation environment. FRONTIERS IN NEUROERGONOMICS 2023; 4:1274730. [PMID: 38234482 PMCID: PMC10790930 DOI: 10.3389/fnrgo.2023.1274730] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 08/08/2023] [Accepted: 10/31/2023] [Indexed: 01/19/2024]
Abstract
Deep reinforcement learning (RL) is used as a strategy to teach robot agents how to autonomously learn complex tasks. While sparsity is a natural way to define a reward in realistic robot scenarios, it provides poor learning signals for the agent, thus making the design of good reward functions challenging. To overcome this challenge learning from human feedback through an implicit brain-computer interface (BCI) is used. We combined a BCI with deep RL for robot training in a 3-D physical realistic simulation environment. In a first study, we compared the feasibility of different electroencephalography (EEG) systems (wet- vs. dry-based electrodes) and its application for automatic classification of perceived errors during a robot task with different machine learning models. In a second study, we compared the performance of the BCI-based deep RL training to feedback explicitly given by participants. Our findings from the first study indicate the use of a high-quality dry-based EEG-system can provide a robust and fast method for automatically assessing robot behavior using a sophisticated convolutional neural network machine learning model. The results of our second study prove that the implicit BCI-based deep RL version in combination with the dry EEG-system can significantly accelerate the learning process in a realistic 3-D robot simulation environment. Performance of the BCI-based trained deep RL model was even comparable to that achieved by the approach with explicit human feedback. Our findings emphasize the usage of BCI-based deep RL methods as a valid alternative in those human-robot applications where no access to cognitive demanding explicit human feedback is available.
Collapse
Affiliation(s)
- Mathias Vukelić
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| | - Michael Bui
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| | - Anna Vorreuther
- Applied Neurocognitive Systems, Institute of Human Factors and Technology Management (IAT), University of Stuttgart, Stuttgart, Germany
| | - Katharina Lingelbach
- Applied Neurocognitive Systems, Fraunhofer Institute for Industrial Engineering (IAO), Stuttgart, Germany
| |
Collapse
|