Adaptive Filtering for Improved EEG-Based Mental Workload Assessment of Ambulant Users

Measuring Bound Attention During Complex Liver Surgery Planning: Feasibility Study

BACKGROUND

The integration of advanced technologies such as augmented reality (AR) and virtual reality (VR) into surgical procedures has garnered significant attention. However, the introduction of these innovations requires thorough evaluation in the context of human-machine interaction. Despite their potential benefits, new technologies can complicate surgical tasks and increase the cognitive load on surgeons, potentially offsetting their intended advantages. It is crucial to evaluate these technologies not only for their functional improvements but also for their impact on the surgeon's workload in clinical settings. A surgical team today must increasingly navigate advanced technologies such as AR and VR, aiming to reduce surgical trauma and enhance patient safety. However, each innovation needs to be evaluated in terms of human-machine interaction. Even if an innovation appears to bring advancements to the field it is applied in, it may complicate the work and increase the surgeon's workload rather than benefiting the surgeon.

OBJECTIVE

This study aims to establish a method for objectively determining the additional workload generated using AR or VR glasses in a clinical context for the first time.

METHODS

Electroencephalography (EEG) signals were recorded using a passive auditory oddball paradigm while 9 participants performed surgical planning for liver resection across 3 different conditions: (1) using AR glasses, (2) VR glasses, and (3) the conventional planning software on a computer.

RESULTS

The electrophysiological results, that is, the potentials evoked by the auditory stimulus, were compared with the subjectively perceived stress of the participants, as determined by the National Aeronautics and Space Administration-Task Load Index (NASA-TLX) questionnaire. The AR condition had the highest scores for mental demand (median 75, IQR 70-85), effort (median 55, IQR 30-65), and frustration (median 40, IQR 15-75) compared with the VR and PC conditions. The analysis of the EEG revealed a trend toward a lower amplitude of the N1 component as well as for the P3 component at the central electrodes in the AR condition, suggesting a higher workload for participants when using AR glasses. In addition, EEG components in the VR condition did not reveal any noticeable differences compared with the EEG components in the conventional planning condition. For the P1 component, the VR condition elicited significantly earlier latencies at the Fz electrode (mean 75.3 ms, SD 25.8 ms) compared with the PC condition (mean 99.4 ms, SD 28.6 ms).

CONCLUSIONS

The results suggest a lower stress level when using VR glasses compared with AR glasses, likely due to the 3D visualization of the liver model. Additionally, the alignment between subjectively determined results and objectively determined results confirms the validity of the study design applied in this research.

An Empirical Model-Based Algorithm for Removing Motion-Caused Artifacts in Motor Imagery EEG Data for Classification Using an Optimized CNN Model

Electroencephalography (EEG) is a non-invasive technique with high temporal resolution and cost-effective, portable, and easy-to-use features. Motor imagery EEG (MI-EEG) data classification is one of the key applications within brain-computer interface (BCI) systems, utilizing EEG signals from motor imagery tasks. BCI is very useful for people with severe mobility issues like quadriplegics, spinal cord injury patients, stroke patients, etc., giving them the freedom to a certain extent to perform activities without the need for a caretaker, like driving a wheelchair. However, motion artifacts can significantly affect the quality of EEG recordings. The conventional EEG enhancement algorithms are effective in removing ocular and muscle artifacts for a stationary subject but not as effective when the subject is in motion, e.g., a wheelchair user. In this research study, we propose an empirical error model-based artifact removal approach for the cross-subject classification of motor imagery (MI) EEG data using a modified CNN-based deep learning algorithm, designed to assist wheelchair users with severe mobility issues. The classification method applies to real tasks with measured EEG data, focusing on accurately interpreting motor imagery signals for practical application. The empirical error model evolved from the inertial sensor-based acceleration data of the subject in motion, the weight of the wheelchair, the weight of the subject, and the surface friction of the terrain under the wheelchair. Three different wheelchairs and five different terrains, including road, brick, concrete, carpet, and marble, are used for artifact data recording. After evaluating and benchmarking the proposed CNN and empirical model, the classification accuracy achieved is 94.04% for distinguishing between four specific classes: left, right, front, and back. This accuracy demonstrates the model's effectiveness compared to other state-of-the-art techniques. The comparative results show that the proposed approach is a potentially effective way to raise the decoding efficiency of motor imagery BCI.

Removal of movement artifacts and assessment of mental stress analyzing electroencephalogram of non-driving passengers under whole-body vibration

The discomfort caused by whole-body vibration (WBV) has long been assessed using subjective surveys or objective measurements of body acceleration. However, surveys have the disadvantage that some of participants often express their feelings in a capricious manner, and acceleration data cannot take into account individual preferences and experiences of their emotions. In this study, we investigated vibration-induced mental stress using the electroencephalogram (EEG) of 22 seated occupants excited by random vibrations. Between the acceleration and the EEG signal, which contains electrical noise due to the head shaking caused by random vibrations, we found that there was a strong correlation, which acts as an artifact in the EEG, and therefore we removed it using an adaptive filter. After removing the artifact, we analyzed the characteristics of the brainwaves using topographic maps and observed that the activities detected in the frontal electrodes showed significant differences between the static and vibration conditions. Further, frontal alpha asymmetry (FAA) and relative band power indices in the frontal electrodes were analyzed statistically to assess mental stress under WBV. As the vibration level increased, EEG analysis in the frontal electrodes showed a decrease in FAA and alpha power but an increase in gamma power. These results are in good agreement with the literature in the sense that FAA and alpha band power decreases with increasing stress, thus demonstrating that WBV causes mental stress and that the stress increases with the vibration level. EEG assessment of stress during WBV is expected to be used in the evaluation of ride comfort alongside existing self-report and acceleration methods.

Blink-Related Oscillations Provide Naturalistic Assessments of Brain Function and Cognitive Workload within Complex Real-World Multitasking Environments

BACKGROUND

There is a significant need to monitor human cognitive performance in complex environments, with one example being pilot performance. However, existing assessments largely focus on subjective experiences (e.g., questionnaires) and the evaluation of behavior (e.g., aircraft handling) as surrogates for cognition or utilize brainwave measures which require artificial setups (e.g., simultaneous auditory stimuli) that intrude on the primary tasks. Blink-related oscillations (BROs) are a recently discovered neural phenomenon associated with spontaneous blinking that can be captured without artificial setups and are also modulated by cognitive loading and the external sensory environment-making them ideal for brain function assessment within complex operational settings.

METHODS

Electroencephalography (EEG) data were recorded from eight adult participants (five F, M = 21.1 years) while they completed the Multi-Attribute Task Battery under three different cognitive loading conditions. BRO responses in time and frequency domains were derived from the EEG data, and comparisons of BRO responses across cognitive loading conditions were undertaken. Simultaneously, assessments of blink behavior were also undertaken.

RESULTS

Blink behavior assessments revealed decreasing blink rate with increasing cognitive load (p < 0.001). Prototypical BRO responses were successfully captured in all participants (p < 0.001). BRO responses reflected differences in task-induced cognitive loading in both time and frequency domains (p < 0.05). Additionally, reduced pre-blink theta band desynchronization with increasing cognitive load was also observed (p < 0.05).

CONCLUSION

This study confirms the ability of BRO responses to capture cognitive loading effects as well as preparatory pre-blink cognitive processes in anticipation of the upcoming blink during a complex multitasking situation. These successful results suggest that blink-related neural processing could be a potential avenue for cognitive state evaluation in operational settings-both specialized environments such as cockpits, space exploration, military units, etc. and everyday situations such as driving, athletics, human-machine interactions, etc.-where human cognition needs to be seamlessly monitored and optimized.

Evidence for modulation of EEG microstates by mental workload levels and task types

Electroencephalography (EEG) microstate analysis has become a popular tool for studying the spatial and temporal dynamics of large-scale electrophysiological activities in the brain in recent years. Four canonical topographies of the electric field (classes A, B, C, and D) have been widely identified, and changes in microstate parameters are associated with several psychiatric disorders and cognitive functions. Recent studies have reported the modulation of EEG microstate by mental workload (MWL). However, the common practice of evaluating MWL is in a specific task. Whether the modulation of microstate by MWL is consistent across different types of tasks is still not clear. Here, we studied the topographies and dynamics of microstate in two independent MWL tasks: NBack and the multi-attribute task battery (MATB) and showed that the modulation of MWL on microstate topographies and parameters depended on tasks. We found that the parameters of microstates A and C, and the topographies of microstates A, B, and D were significantly different between the two tasks. Meanwhile, all four microstate topographies and parameters of microstates A and C were different during the NBack task, but no significant difference was found during the MATB task. Furthermore, we employed a support vector machine recursive feature elimination procedure to investigate whether microstate parameters were suitable for MWL classification. An averaged classification accuracy of 87% for within-task and 78% for cross-task MWL discrimination was achieved with at least 10 features. Collectively, our findings suggest that topographies and parameters of microstates can provide valuable information about neural activity patterns with a dynamic temporal structure at different levels of MWL, but the modulation of MWL depends on tasks and their corresponding functional systems. Moreover, as a potential indicator, microstate parameters could be used to distinguish MWL.

Mental fatigue assessment by an arbitrary channel EEG based on morphological features and LSTM-CNN

In order to achieve more sensitive mental fatigue assessment (MFA) based on an arbitrary channel EEG, this study proposed a series of feature extraction methods that combine mathematical morphology (MM), as well as an LSTM-CNN architecture. Firstly, 37 subjects had their resting-state EEGs collected at rested wakefulness (RW) and after 24 h of sleep deprivation (SD) using a 30-channel EEG acquisition device, the RW and SD groups were regarded as the negative and positive groups of mental fatigue, respectively, and the EEG collection were further categorized into two conditions: eye-opened state (EO) and eye-closed state (EC). Then, since MM can reflect the morphological characteristics of EEG rhythms and their potentials relatively independently of the time-frequency analysis and phase calculation, the MM methods were found to better reflect the mental fatigue after SD statistically, whether for single features (ANOVA: p<0.000001), multiple features (clustering by K-means, t-test: p<0.01), or time series feature spaces (calculating CD, t-test: p<0.01) of a single channel. Finally, the LSTM-CNN enhanced the generalization ability when dealing with different single-channel EEG by combining GRUs with convolutional layers: comparing the AUCs of different architectures for MFA based on an arbitrary channel, LSTM-CNN (0.992) > LSTM network (0.94) > CNN (0.831) > MLP (0.754). Moreover, the use of MM also improved the accuracy of analyzed architectures, and the true/false positive rate (TPR/FPR) of the LSTM-CNN architecture for MFA based on an arbitrary channel reached 97.024 %/3.497 %, which provided a feasible solution for the arbitrary channel EEG-based MFA.

Methods for motion artifact reduction in online brain-computer interface experiments: a systematic review

Brain-computer interfaces (BCIs) have emerged as a promising technology for enhancing communication between the human brain and external devices. Electroencephalography (EEG) is particularly promising in this regard because it has high temporal resolution and can be easily worn on the head in everyday life. However, motion artifacts caused by muscle activity, fasciculation, cable swings, or magnetic induction pose significant challenges in real-world BCI applications. In this paper, we present a systematic review of methods for motion artifact reduction in online BCI experiments. Using the PRISMA filter method, we conducted a comprehensive literature search on PubMed, focusing on open access publications from 1966 to 2022. We evaluated 2,333 publications based on predefined filtering rules to identify existing methods and pipelines for motion artifact reduction in EEG data. We present a lookup table of all papers that passed the defined filters, all used methods, and pipelines and compare their overall performance and suitability for online BCI experiments. We summarize suitable methods, algorithms, and concepts for motion artifact reduction in online BCI applications, highlight potential research gaps, and discuss existing community consensus. This review aims to provide a comprehensive overview of the current state of the field and guide researchers in selecting appropriate methods for motion artifact reduction in online BCI experiments.

Quantifying time perception during virtual reality gameplay using a multimodal biosensor-instrumented headset: a feasibility study

We have all experienced the sense of time slowing down when we are bored or speeding up when we are focused, engaged, or excited about a task. In virtual reality (VR), perception of time can be a key aspect related to flow, immersion, engagement, and ultimately, to overall quality of experience. While several studies have explored changes in time perception using questionnaires, limited studies have attempted to characterize them objectively. In this paper, we propose the use of a multimodal biosensor-embedded VR headset capable of measuring electroencephalography (EEG), electrooculography (EOG), electrocardiography (ECG), and head movement data while the user is immersed in a virtual environment. Eight gamers were recruited to play a commercial action game comprised of puzzle-solving tasks and first-person shooting and combat. After gameplay, ratings were given across multiple dimensions, including (1) the perception of time flowing differently than usual and (2) the gamers losing sense of time. Several features were extracted from the biosignals, ranked based on a two-step feature selection procedure, and then mapped to a predicted time perception rating using a Gaussian process regressor. Top features were found to come from the four signal modalities and the two regressors, one for each time perception scale, were shown to achieve results significantly better than chance. An in-depth analysis of the top features is presented with the hope that the insights can be used to inform the design of more engaging and immersive VR experiences.

A simplified design of a cEEGrid ear-electrode adapter for the OpenBCI biosensing platform

We present a simplified design of an ear-centered sensing system built around the OpenBCI Cyton & Daisy biosignal amplifiers and the flex-printed cEEGrid ear-EEG electrodes. This design reduces the number of components that need to be sourced, reduces mechanical artefacts on the recording data through better cable placement, and simplifies the assembly. Besides describing how to replicate and use the system, we highlight promising application scenarios, particularly the observation of large-amplitude activity patterns (e.g., facial muscle activities) and frequency-band neural activity (e.g., alpha and beta band power modulations for mental workload detection). Further, examples for common measurement artefacts and methods for removing them are provided, introducing a prototypical application of adaptive filters to this system. Lastly, as a promising use case, we present findings from a single-user study that highlights the system's capability of detecting jaw clenching events robustly when contrasted with 26 other facial activities. Thereby, the system could, for instance, be used to devise applications that reduce pathological jaw clenching and teeth grinding (bruxism). These findings underline that the system represents a valuable prototyping platform for advancing ear-based electrophysiological sensing systems and a low-cost alternative to current commercial alternatives.

The brain under cognitive workload: Neural networks underlying multitasking performance in the multi-attribute task battery

Multitasking is a common requirement in many occupations. Considerable research has demonstrated that performance declines as a result of multitasking, and that it engages multiple brain regions. Despite growing evidence suggesting that brain regions operate as networks, minimal research has investigated the cognitive brain networks implicated in multitasking. The Multi-Attribute Task Battery II (MATB) is a common method for assessing multitasking ability that simulates a pilot's operational environment inside an aircraft cockpit. The aim of the present study was to examine multitasking performance on the MATB, and the associated neural patterns underlying performance with functional magnetic resonance imaging (fMRI). Twenty-four participants completed the MATB in the fMRI scanner. Participants completed four runs of the MATB in a 2 (Task: multitasking vs. single tasking) × 2 (Difficulty: hard vs. easy) design. MATB performance was measured as a function of accuracy. We analyzed the fMRI brain scans using both static and dynamic functional connectivity to determine whether there were differences in the connectivity patterns associated with each of the four conditions. A significant interaction between Task and Difficulty was observed such that multitasking performance accuracy, which was derived from the average across tasks, was lower than single tasking in the hard, but not easy, condition. The fMRI data revealed that static and dynamic functional connectivity between the default mode and dorsal attention networks was stronger during multitasking relative to single tasking. The static functional connectivity between the default mode and left frontoparietal networks, along with the dynamic functional connectivity between the dorsal attention and left frontoparietal networks, were both more anti-correlated during multitasking relative to single tasking. Taken together, the static and dynamic functional connectivity analyses provide complementary information to reveal the interactions among cognitive networks that support multitasking performance. Targeting these networks may offer a path to enhance multitasking ability through the application of neurostimulation and neuroenhancement techniques.

Modulation Spectral Signal Representation for Quality Measurement and Enhancement of Wearable Device Data: A Technical Note

Wearable devices are burgeoning, and applications across numerous verticals are emerging, including human performance monitoring, at-home patient monitoring, and health tracking, to name a few. Off-the-shelf wearables have been developed with focus on portability, usability, and low-cost. As such, when deployed in highly ecological settings, wearable data can be corrupted by artifacts and by missing data, thus severely hampering performance. In this technical note, we overview a signal processing representation called the modulation spectrum. The representation quantifies the rate-of-change of different spectral magnitude components and is shown to separate signal from noise, thus allowing for improved quality measurement, quality enhancement, and noise-robust feature extraction, as well as for disease characterization. We provide an overview of numerous applications developed by the authors over the last decade spanning different wearable modalities and list the results obtained from experimental results alongside comparisons with various state-of-the-art benchmark methods. Open-source software is showcased with the hope that new applications can be developed. We conclude with a discussion on possible future research directions, such as context awareness, signal compression, and improved input representations for deep learning algorithms.

Immersive media experience: a survey of existing methods and tools for human influential factors assessment

Virtual reality (VR) applications, especially those where the user is untethered to a computer, are becoming more prevalent as new hardware is developed, computational power and artificial intelligence algorithms are available, and wireless communication networks are becoming more reliable, fast, and providing higher reliability. In fact, recent projections show that by 2022 the number of VR users will double, suggesting the sector was not negatively affected by the worldwide COVID-19 pandemic. The success of any immersive communication system is heavily dependent on the user experience it delivers, thus now more than ever has it become crucial to develop reliable models of immersive media experience (IMEx). In this paper, we survey the literature for existing methods and tools to assess human influential factors (HIFs) related to IMEx. In particular, subjective, behavioural, and psycho-physiological methods are covered. We describe tools available to monitor these HIFs, including the user's sense of presence and immersion, cybersickness, and mental/affective states, as well as their role in overall experience. Special focus is placed on psycho-physiological methods, as it was found that such in-depth evaluation was lacking from the existing literature. We conclude by touching on emerging applications involving multiple-sensorial immersive media and provide suggestions for future research directions to fill existing gaps. It is hoped that this survey will be useful for researchers interested in building new immersive (adaptive) applications that maximize user experience.

EEG-TNet: An End-To-End Brain Computer Interface Framework for Mental Workload Estimation

The mental workload (MWL) of different occupational groups' workers is the main and direct factor of unsafe behavior, which may cause serious accidents. One of the new and useful technologies to estimate MWL is the Brain computer interface (BCI) based on EEG signals, which is regarded as the gold standard of cognitive status. However, estimation systems involving handcrafted EEG features are time-consuming and unsuitable to apply in real-time. The purpose of this study was to propose an end-to-end BCI framework for MWL estimation. First, a new automated data preprocessing method was proposed to remove the artifact without human interference. Then a new neural network structure named EEG-TNet was designed to extract both the temporal and frequency information from the original EEG. Furthermore, two types of experiments and ablation studies were performed to prove the effectiveness of this model. In the subject-dependent experiment, the estimation accuracy of dual-task estimation (No task vs. TASK) and triple-task estimation (Lo vs. Mi vs. Hi) reached 99.82 and 99.21%, respectively. In contrast, the accuracy of different tasks reached 82.78 and 66.83% in subject-independent experiments. Additionally, the ablation studies proved that preprocessing method and network structure had significant contributions to estimation MWL. The proposed method is convenient without any human intervention and outperforms other related studies, which becomes an effective way to reduce human factor risks.

Affective Neurofeedback Under Naturalistic Conditions: A Mini-Review of Current Achievements and Open Challenges

Affective neurofeedback training allows for the self-regulation of the putative circuits of emotion regulation. This approach has recently been studied as a possible additional treatment for psychiatric disorders, presenting positive effects in symptoms and behaviors. After neurofeedback training, a critical aspect is the transference of the learned self-regulation strategies to outside the laboratory and how to continue reinforcing these strategies in non-controlled environments. In this mini-review, we discuss the current achievements of affective neurofeedback under naturalistic setups. For this, we first provide a brief overview of the state-of-the-art for affective neurofeedback protocols. We then discuss virtual reality as a transitional step toward the final goal of "in-the-wild" protocols and current advances using mobile neurotechnology. Finally, we provide a discussion of open challenges for affective neurofeedback protocols in-the-wild, including topics such as convenience and reliability, environmental effects in attention and workload, among others.