1
|
Demirezen G, Taşkaya Temizel T, Brouwer AM. Reproducible machine learning research in mental workload classification using EEG. FRONTIERS IN NEUROERGONOMICS 2024; 5:1346794. [PMID: 38660590 PMCID: PMC11039816 DOI: 10.3389/fnrgo.2024.1346794] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 11/29/2023] [Accepted: 03/22/2024] [Indexed: 04/26/2024]
Abstract
This study addresses concerns about reproducibility in scientific research, focusing on the use of electroencephalography (EEG) and machine learning to estimate mental workload. We established guidelines for reproducible machine learning research using EEG and used these to assess the current state of reproducibility in mental workload modeling. We first started by summarizing the current state of reproducibility efforts in machine learning and in EEG. Next, we performed a systematic literature review on Scopus, Web of Science, ACM Digital Library, and Pubmed databases to find studies about reproducibility in mental workload prediction using EEG. All of this previous work was used to formulate guidelines, which we structured along the widely recognized Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. By using these guidelines, researchers can ensure transparency and comprehensiveness of their methodologies, therewith enhancing collaboration and knowledge-sharing within the scientific community, and enhancing the reliability, usability and significance of EEG and machine learning techniques in general. A second systematic literature review extracted machine learning studies that used EEG to estimate mental workload. We evaluated the reproducibility status of these studies using our guidelines. We highlight areas studied and overlooked and identify current challenges for reproducibility. Our main findings include limitations on reporting performance on unseen test data, open sharing of data and code, and reporting of resources essential for training and inference processes.
Collapse
Affiliation(s)
- Güliz Demirezen
- Department of Information Systems, Graduate School of Informatics, Middle East Technical University, Ankara, Türkiye
| | - Tuğba Taşkaya Temizel
- Department of Data Informatics, Graduate School of Informatics, Middle East Technical University, Ankara, Türkiye
| | - Anne-Marie Brouwer
- Human Performance, Netherlands Organisation for Applied Scientific Research (TNO), Soesterberg, Netherlands
- Donders Institute for Brain, Cognition and Behaviour, Radboud University, Nijmegen, Netherlands
| |
Collapse
|
2
|
Mastropietro A, Pirovano I, Marciano A, Porcelli S, Rizzo G. Reliability of Mental Workload Index Assessed by EEG with Different Electrode Configurations and Signal Pre-Processing Pipelines. SENSORS (BASEL, SWITZERLAND) 2023; 23:1367. [PMID: 36772409 PMCID: PMC9920504 DOI: 10.3390/s23031367] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 12/21/2022] [Revised: 01/18/2023] [Accepted: 01/21/2023] [Indexed: 06/18/2023]
Abstract
BACKGROUND AND OBJECTIVE Mental workload (MWL) is a relevant construct involved in all cognitively demanding activities, and its assessment is an important goal in many research fields. This paper aims at evaluating the reproducibility and sensitivity of MWL assessment from EEG signals considering the effects of different electrode configurations and pre-processing pipelines (PPPs). METHODS Thirteen young healthy adults were enrolled and were asked to perform 45 min of Simon's task to elicit a cognitive demand. EEG data were collected using a 32-channel system with different electrode configurations (fronto-parietal; Fz and Pz; Cz) and analyzed using different PPPs, from the simplest bandpass filtering to the combination of filtering, Artifact Subspace Reconstruction (ASR) and Independent Component Analysis (ICA). The reproducibility of MWL indexes estimation and the sensitivity of their changes were assessed using Intraclass Correlation Coefficient and statistical analysis. RESULTS MWL assessed with different PPPs showed reliability ranging from good to very good in most of the electrode configurations (average consistency > 0.87 and average absolute agreement > 0.92). Larger fronto-parietal electrode configurations, albeit being more affected by the choice of PPPs, provide better sensitivity in the detection of MWL changes if compared to a single-electrode configuration (18 vs. 10 statistically significant differences detected, respectively). CONCLUSIONS The most complex PPPs have been proven to ensure good reliability (>0.90) and sensitivity in all experimental conditions. In conclusion, we propose to use at least a two-electrode configuration (Fz and Pz) and complex PPPs including at least the ICA algorithm (even better including ASR) to mitigate artifacts and obtain reliable and sensitive MWL assessment during cognitive tasks.
Collapse
Affiliation(s)
- Alfonso Mastropietro
- Institute of Biomedical Technologies, National Research Council, Via Fratelli Cervi 93, 20054 Segrate, Italy
| | - Ileana Pirovano
- Institute of Biomedical Technologies, National Research Council, Via Fratelli Cervi 93, 20054 Segrate, Italy
| | - Alessio Marciano
- Department of Molecular Medicine, University of Pavia, Via Forlanini 6, 27100 Pavia, Italy
| | - Simone Porcelli
- Department of Molecular Medicine, University of Pavia, Via Forlanini 6, 27100 Pavia, Italy
| | - Giovanna Rizzo
- Institute of Biomedical Technologies, National Research Council, Via Fratelli Cervi 93, 20054 Segrate, Italy
| |
Collapse
|
3
|
Pütz S, Rick V, Mertens A, Nitsch V. Using IoT devices for sensor-based monitoring of employees' mental workload: Investigating managers' expectations and concerns. APPLIED ERGONOMICS 2022; 102:103739. [PMID: 35279467 DOI: 10.1016/j.apergo.2022.103739] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Received: 12/13/2021] [Revised: 02/24/2022] [Accepted: 03/01/2022] [Indexed: 06/14/2023]
Abstract
Although the objective assessment of mental workload has been a focus of human factors research, few studies have investigated stakeholders' attitudes towards its implementation in real workplaces. The present study addresses this research gap by surveying N = 702 managers in three European countries (Germany, United Kingdom, Spain) about their expectations and concerns regarding sensor-based monitoring of employee mental workload. The data confirm the relevance of expectations regarding improvements of workplace design and employee well-being, as well as concerns about restrictions of employees' privacy and sovereignty, for the implementation of workload monitoring. Furthermore, Bayesian regression models show that the examined expectations have a substantial positive association with managers' willingness to support workload monitoring in their company. Privacy concerns are identified as a significant barrier to the acceptance of workload monitoring, both in terms of their prevalence among managers and their strong negative relationship with monitoring support.
Collapse
Affiliation(s)
- Sebastian Pütz
- Institute of Industrial Engineering and Ergonomics, RWTH Aachen University, Eilfschornsteinstr. 18, 52062, Aachen, Germany.
| | - Vera Rick
- Institute of Industrial Engineering and Ergonomics, RWTH Aachen University, Eilfschornsteinstr. 18, 52062, Aachen, Germany
| | - Alexander Mertens
- Institute of Industrial Engineering and Ergonomics, RWTH Aachen University, Eilfschornsteinstr. 18, 52062, Aachen, Germany
| | - Verena Nitsch
- Institute of Industrial Engineering and Ergonomics, RWTH Aachen University, Eilfschornsteinstr. 18, 52062, Aachen, Germany; Fraunhofer Institute for Communication, Information Processing and Ergonomics FKIE, Campus-Boulevard 55-57, 52074, Aachen, Germany
| |
Collapse
|
4
|
Hamann A, Carstengerdes N. Investigating mental workload-induced changes in cortical oxygenation and frontal theta activity during simulated flights. Sci Rep 2022; 12:6449. [PMID: 35440733 PMCID: PMC9018717 DOI: 10.1038/s41598-022-10044-y] [Citation(s) in RCA: 8] [Impact Index Per Article: 4.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/20/2021] [Accepted: 03/30/2022] [Indexed: 11/09/2022] Open
Abstract
Monitoring pilots' cognitive states becomes increasingly important in aviation. Physiological measurement can detect increased mental workload (MWL) even before performance declines. Yet, changes in MWL are rarely varied systematically and few studies control for confounding effects of other cognitive states. The present study targets these shortcomings by analysing the effects of stepwise increased MWL on cortical activation, while controlling for mental fatigue (MF). 35 participants conducted a simulated flight with an incorporated adapted n-back and monitoring task. We recorded cortical activation with concurrent EEG and fNIRS measurement, performance, self-reported MWL and MF. Our results show the successful manipulation of MWL without confounding effects of MF. Higher task difficulty elicited higher subjective MWL ratings, performance decline, higher frontal theta activity and reduced frontal deoxyhaemoglobin (Hbr) concentration. Using both EEG and fNIRS, we could discriminate all induced MWL levels. fNIRS was more sensitive to tasks with low difficulty, and EEG to tasks with high difficulty. Our findings further suggest a plateau effect for high MWL that could present an upper boundary to individual cognitive capacity. Our results highlight the benefits of physiological measurement in aviation, both for assessment of cognitive states and as a data source for adaptive assistance systems.
Collapse
Affiliation(s)
- Anneke Hamann
- Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Institut für Flugführung, Lilienthalplatz 7, 38108, Braunschweig, Germany.
| | - Nils Carstengerdes
- Deutsches Zentrum für Luft- und Raumfahrt e.V. (DLR), Institut für Flugführung, Lilienthalplatz 7, 38108, Braunschweig, Germany
| |
Collapse
|
5
|
Zhong JY, Goh SK, Woo CJ, Alam S. Impact of Spatial Orientation Ability on Air Traffic Conflict Detection in a Simulated Free Route Airspace Environment. Front Hum Neurosci 2022; 16:739866. [PMID: 35463929 PMCID: PMC9024046 DOI: 10.3389/fnhum.2022.739866] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 07/12/2021] [Accepted: 03/02/2022] [Indexed: 11/19/2022] Open
Abstract
In the selection of job candidates who have the mental ability to become professional ATCOs, psychometric testing has been a ubiquitous activity in the ATM domain. To contribute to psychometric research in the ATM domain, we investigated the extent to which spatial orientation ability (SOA), as conceptualized in the spatial cognition and navigation literature, predicted air traffic conflict detection performance in a simulated free route airspace (FRA) environment. The implementation of free route airspace (FRA) over the past few years, notably in Europe, have facilitated air traffic services by giving greater flexibility to aviation operators in planning and choosing preferred air routes that can lead to quicker arrivals. FRA offers enhanced system safety and efficiency, but these benefits can be outweighed by the introduction of air traffic conflicts that are geometrically more complex. Such conflicts can arise from increased number and distribution of conflict points, as well as from elevated uncertainty in aircraft maneuvering (for instance, during heading changes). Overall, these issues will make conflict detection more challenging for air traffic controllers (ATCOs). Consequently, there is a need to select ATCOs with suitably high levels of spatial orientation ability (SOA) to ensure flight safety under FRA implementation. In this study, we tested 20 participants who are eligible for ATCO job application, and found that response time-based performance on a newly developed, open access, computerized spatial orientation test (SOT) predicted time to loss of minimum separation (tLMS) performance on an air traffic conflict detection task (AT-CDT) we designed. We found this predictive relationship to be significant to a moderately large extent under scenarios with high air traffic density (raw regression coefficient = 0.58). Moreover, we demonstrated our AT-CDT as a valid test in terms of eliciting well-known mental workload and spatial learning effects. We explained these findings in light of similar or overlapping mental processes that were most likely activated optimally under task conditions featuring approximately equal numbers of outcome-relevant stimuli. We conclude by discussing the further application of the SOT to the selection of prospective ATCOs who can demonstrate high levels of conflict detection performance in FRA during training simulations.
Collapse
Affiliation(s)
- Jimmy Y. Zhong
- Air Traffic Management Research Institute (ATMRI), School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore, Singapore
- Office of Education Research, National Institute of Education, Nanyang Technological University, Singapore, Singapore
- *Correspondence: Jimmy Y. Zhong
| | - Sim Kuan Goh
- Air Traffic Management Research Institute (ATMRI), School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore, Singapore
- School of Electrical Engineering and Artificial Intelligence, Xiamen University, Selangor, Malaysia
| | - Chuan Jie Woo
- Air Traffic Management Research Institute (ATMRI), School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore, Singapore
| | - Sameer Alam
- Air Traffic Management Research Institute (ATMRI), School of Mechanical and Aerospace Engineering, Nanyang Technological University, Singapore, Singapore
| |
Collapse
|
6
|
Grenzebach J, Romanus E. Quantifying the Effect of Noise on Cognitive Processes: A Review of Psychophysiological Correlates of Workload. Noise Health 2022; 24:199-214. [PMID: 36537445 PMCID: PMC10088430 DOI: 10.4103/nah.nah_34_22] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/23/2022] Open
Abstract
Noise is present in most work environments, including emissions from machines and devices, irrelevant speech from colleagues, and traffic noise. Although it is generally accepted that noise below the permissible exposure limits does not pose a considerable risk for auditory effects like hearing impairments. Yet, noise can have a direct adverse effect on cognitive performance (non-auditory effects like workload or stress). Under certain circumstances, the observable performance for a task carried out in silence compared to noisy surroundings may not differ. One possible explanation for this phenomenon needs further investigation: individuals may invest additional cognitive resources to overcome the distraction from irrelevant auditory stimulation. Recent developments in measurements of psychophysiological correlates and analysis methods of load-related parameters can shed light on this complex interaction. These objective measurements complement subjective self-report of perceived effort by quantifying unnoticed noise-related cognitive workload. In this review, literature databases were searched for peer-reviewed journal articles that deal with an at least partially irrelevant "auditory stimulation" during an ongoing "cognitive task" that is accompanied by "psychophysiological correlates" to quantify the "momentary workload." The spectrum of assessed types of "auditory stimulations" extended from speech stimuli (varying intelligibility), oddball sounds (repeating short tone sequences), and auditory stressors (white noise, task-irrelevant real-life sounds). The type of "auditory stimulation" was related (speech stimuli) or unrelated (oddball, auditory stressor) to the type of primary "cognitive task." The types of "cognitive tasks" include speech-related tasks, fundamental psychological assessment tasks, and real-world/simulated tasks. The "psychophysiological correlates" include pupillometry and eye-tracking, recordings of brain activity (hemodynamic, potentials), cardiovascular markers, skin conductance, endocrinological markers, and behavioral markers. The prevention of negative effects on health by unexpected stressful soundscapes during mental work starts with the continuous estimation of cognitive workload triggered by auditory noise. This review gives a comprehensive overview of methods that were tested for their sensitivity as markers of workload in various auditory settings during cognitive processing.
Collapse
|
7
|
Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems. COMPUTERS 2021. [DOI: 10.3390/computers10060081] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains a high level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The results from the online validation of the ANFIS models demonstrated that five of the ANFIS models (containing different feature combinations of eye activity and control input features) all demonstrated good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be seen with the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all the ANFIS models tested. Hence, this study demonstrates the ability for online multimodal fusion from features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL.
Collapse
|