Continuous decoding of cognitive load from electroencephalography reveals task-general and task-specific correlates

Evaluation of mental load using EEG and eye movement characteristics

Mental load is a major cause of human-induced accidents. In this study, an explosive impact sensitivity experiment was used to induce mental load. A combination of subjective questionnaires and objective prospective time-distance tests were used to judge whether subjects experienced mental load. Four indicators, namely, β, γ, mean pupil diameter, and fixation time were selected by statistical analysis and PCA for the construction of a mental load assessment model. The study found that the occipital lobe was the most sensitive to mental load, especially β and γ bands. Lastly, it was found that subjects showed different degrees of mental load for the same mental load induction task. The results of the study are applicable to the evaluation and monitoring of the mental characteristics of workers and provide a scientific basis for adjusting the mental load of workers over time to reduce the rate of accidents and enhance production efficiency.

Reproducible machine learning research in mental workload classification using EEG

This study addresses concerns about reproducibility in scientific research, focusing on the use of electroencephalography (EEG) and machine learning to estimate mental workload. We established guidelines for reproducible machine learning research using EEG and used these to assess the current state of reproducibility in mental workload modeling. We first started by summarizing the current state of reproducibility efforts in machine learning and in EEG. Next, we performed a systematic literature review on Scopus, Web of Science, ACM Digital Library, and Pubmed databases to find studies about reproducibility in mental workload prediction using EEG. All of this previous work was used to formulate guidelines, which we structured along the widely recognized Cross-Industry Standard Process for Data Mining (CRISP-DM) framework. By using these guidelines, researchers can ensure transparency and comprehensiveness of their methodologies, therewith enhancing collaboration and knowledge-sharing within the scientific community, and enhancing the reliability, usability and significance of EEG and machine learning techniques in general. A second systematic literature review extracted machine learning studies that used EEG to estimate mental workload. We evaluated the reproducibility status of these studies using our guidelines. We highlight areas studied and overlooked and identify current challenges for reproducibility. Our main findings include limitations on reporting performance on unseen test data, open sharing of data and code, and reporting of resources essential for training and inference processes.

Temporal and electroencephalography dynamics of surreal marketing

Event-related spectral perturbation analysis was employed in this study to explore whether surreal image designs containing metaphors could influence product marketing effects, including consumers' product curiosity, product comprehension, product preference, and purchase intention. A total of 30 healthy participants aged 21-30 years were recruited. Neurophysiological findings revealed that lower gamma, beta, and theta spectral powers were evoked in the right insula (Brodmann Area 13) by surreal marketing images. This was associated, behaviorally, with the manifestation of higher product curiosity and purchase intention. Based on previous research, the brain functions of this area include novelty, puzzle-solving, and cravings for reward caused by cognitive overload.

Semantic Scene-Object Consistency Modulates N300/400 EEG Components, but Does Not Automatically Facilitate Object Representations

During natural vision, objects rarely appear in isolation, but often within a semantically related scene context. Previous studies reported that semantic consistency between objects and scenes facilitates object perception and that scene-object consistency is reflected in changes in the N300 and N400 components in EEG recordings. Here, we investigate whether these N300/400 differences are indicative of changes in the cortical representation of objects. In two experiments, we recorded EEG signals, while participants viewed semantically consistent or inconsistent objects within a scene; in Experiment 1, these objects were task-irrelevant, while in Experiment 2, they were directly relevant for behavior. In both experiments, we found reliable and comparable N300/400 differences between consistent and inconsistent scene-object combinations. To probe the quality of object representations, we performed multivariate classification analyses, in which we decoded the category of the objects contained in the scene. In Experiment 1, in which the objects were not task-relevant, object category could be decoded from ~100 ms after the object presentation, but no difference in decoding performance was found between consistent and inconsistent objects. In contrast, when the objects were task-relevant in Experiment 2, we found enhanced decoding of semantically consistent, compared with semantically inconsistent, objects. These results show that differences in N300/400 components related to scene-object consistency do not index changes in cortical object representations but rather reflect a generic marker of semantic violations. Furthermore, our findings suggest that facilitatory effects between objects and scenes are task-dependent rather than automatic.

3D Space Layout Design of Holographic Command Cabin Information Display in Mixed Reality Environment Based on HoloLens 2

After the command and control information of the command and control cabin is displayed in the form of mixed reality, the large amount of real-time information and static information contained in it will form a dynamic situation that changes all the time. This brings a great burden to the system operator's cognition, decision-making and operation. In order to solve this problem, this paper studies the three-dimensional spatial layout of holographic command cabin information display in a mixed reality environment. A total of 15 people participated in the experiment, of which 10 were the subjects of the experiment and 5 were the staff of the auxiliary experiment. Ten subjects used the HoloLens 2 generation to conduct visual characteristics and cognitive load experiments and collected and analyzed the subjects’ task completion time, error rate, eye movement and EEG and subjective evaluation data. Through the analysis of experimental data, the laws of visual and cognitive features of three-dimensional space in a mixed reality environment can be obtained. This paper systematically explores the effects of three key attributes: depth distance, information layer number and target relative position depth distance of information distribution in a 3D space, on visual search performance and on cognitive load. The experimental results showed that the optimal depth distance range for information display in the mixed reality environment is: the best depth distance for operation interactions (0.6 m~1.0 m), the best depth distance for accurate identification (2.4 m~2.8 m) and the overall situational awareness best-in-class depth distance (3.4 m~3.6 m). Under a certain angle of view, the number of information layers in the space is as small as possible, and the number of information layers should not exceed five at most. The relative position depth distance between the information layers in space ranges from 0.2 m to 0.35 m. Based on this theory, information layout in a 3D space can achieve a faster and more accurate visual search in a mixed reality environment and effectively reduce the cognitive load.

Optimization and Improvement of Display Interaction System Based on Complex Command and Control Tasks

A complex command and control task was selected as the test task, which included observing the overall and local situation, the interactive operation and situation display of detection equipment, the erection and launch of air defense equipment, and the check and display status. The disadvantages of the traditional two-dimensional display interactive system include poor intuitiveness, insufficient information display dimension and complicated interactive operation. The mixed reality display interaction system can avoid these problems well and has the advantages of good portability and high efficiency, but this display interaction system has the problem of high cognitive load. Therefore, based on the premise of completing the same complex task, how to select and improve the display interaction system has become a problem worthy of urgent research. Based on the same complex command and control task, this paper compared the traditional two-dimensional display interaction system and the mixed reality display interaction system and analyzed the performance and cognitive load of the two systems. It is concluded that when completing the same task, the performance of the mixed reality display interaction system is significantly higher than that of the traditional two-dimensional display interaction system, but the cognitive load is slightly higher than that of the traditional two-dimensional display. Cognitive load was reduced while task performance was improved through multi-channel improvements to the mixed reality display interaction system. Considering the effects of performance and cognitive load, the improved multi-channel mixed reality display interaction system is superior to the unimproved mixed reality display interaction system and the two-dimensional display interaction system. This research provides an improvement strategy for the existing display interaction system and provides a new display interaction mode for future aerospace equipment and multi-target, multi-dimensional command and control tasks in war.

Promise for Personalized Diagnosis? Assessing the Precision of Wireless Consumer-Grade Electroencephalography across Mental States

In the last decade there has been significant growth in the interest and application of using EEG (electroencephalography) outside of laboratory as well as in medical and clinical settings, for more ecological and mobile applications. However, for now such applications have mainly included military, educational, cognitive enhancement, and consumer-based games. Given the monetary and ecological advantages, consumer-grade EEG devices such as the Emotiv EPOC have emerged, however consumer-grade devices make certain compromises of data quality in order to become affordable and easy to use. The goal of this study was to investigate the reliability and accuracy of EPOC as compared to a research-grade device, Brainvision. To this end, we collected data from participants using both devices during three distinct cognitive tasks designed to elicit changes in arousal, valence, and cognitive load: namely, Affective Norms for English Words, International Affective Picture System, and the n-Back task. Our design and analytical strategies followed an ideographic person-level approach (electrode-wise analysis of vincentized repeated measures). We aimed to assess how well the Emotiv could differentiate between mental states using an Event-Related Band Power approach and EEG features such as amplitude and power, as compared to Brainvision. The Emotiv device was able to differentiate mental states during these tasks to some degree, however it was generally poorer than Brainvision, with smaller effect sizes. The Emotiv may be used with reasonable reliability and accuracy in ecological settings and in some clinical contexts (for example, for training professionals), however Brainvision or other, equivalent research-grade devices are still recommended for laboratory or medical based applications.

The Task Pre-Configuration Is Associated With Cognitive Performance Evidence From the Brain Synchrony

Although many resting state and task state characteristics have been studied, it is still unclear how the brain network switches from the resting state during tasks. The current theory shows that the brain is a complex dynamic system and synchrony is defined to measure brain activity. The study compared the changes of synchrony between the resting state and different task states in healthy young participants (N = 954). It also examined the ability to switch from the resting state to the task-general architecture of synchrony. We found that the synchrony increased significantly during the tasks. And the results showed that the brain has a task-general architecture of synchrony during different tasks. The main feature of task-based reasoning is that the increase in synchrony of high-order cognitive networks is significant, while the increase in synchrony of sensorimotor networks is relatively low. In addition, the high synchrony of high-order cognitive networks in the resting state can promote task switching effectively and the pre-configured participants have better cognitive performance, which shows that spontaneous brain activity and cognitive ability are closely related. These results revealed changes in the brain network configuration for switching between the resting state and task state, highlighting the consistent changes in the brain network between different tasks. Also, there was an important relationship between the switching ability and the cognitive performance.

Gaze dynamics are sensitive to target orienting for working memory encoding in virtual reality

Numerous studies have demonstrated that visuospatial attention is a requirement for successful working memory encoding. It is unknown, however, whether this established relationship manifests in consistent gaze dynamics as people orient their visuospatial attention toward an encoding target when searching for information in naturalistic environments. To test this hypothesis, participants' eye movements were recorded while they searched for and encoded objects in a virtual apartment (Experiment 1). We decomposed gaze into 61 features that capture gaze dynamics and a trained sliding window logistic regression model that has potential for use in real-time systems to predict when participants found target objects for working memory encoding. A model trained on group data successfully predicted when people oriented to a target for encoding for the trained task (Experiment 1) and for a novel task (Experiment 2), where a new set of participants found objects and encoded an associated nonword in a cluttered virtual kitchen. Six of these features were predictive of target orienting for encoding, even during the novel task, including decreased distances between subsequent fixation/saccade events, increased fixation probabilities, and slower saccade decelerations before encoding. This suggests that as people orient toward a target to encode new information at the end of search, they decrease task-irrelevant, exploratory sampling behaviors. This behavior was common across the two studies. Together, this research demonstrates how gaze dynamics can be used to capture target orienting for working memory encoding and has implications for real-world use in technology and special populations.

Cross-Task Consistency of Electroencephalography-Based Mental Workload Indicators: Comparisons Between Power Spectral Density and Task-Irrelevant Auditory Event-Related Potentials

Mental workload (MWL) estimators based on ongoing electroencephalography (EEG) and event-related potentials (ERPs) have shown great potentials to build adaptive aiding systems for human-machine systems by estimating MWL in real time. However, extracting EEG features which are consistent in indicating MWL across different tasks is still one of the critical challenges. This study attempts to compare the cross-task consistency in indexing MWL variations between two commonly used EEG-based MWL indicators, power spectral density (PSD) of ongoing EEG and task-irrelevant auditory ERPs (tir-aERPs). The verbal N-back and the multi-attribute task battery (MATB), both with two difficulty levels, were employed in the experiment, along with task-irrelevant auditory probes. EEG was recorded from 17 subjects when they were performing the tasks. The tir-aERPs elicited by the auditory probes and the relative PSDs of ongoing EEG between two consecutive auditory probes were extracted and statistically analyzed to reveal the effects of MWL and task type. Discriminant analysis and support vector machine were employed to examine the generalization of tir-aERP and PSD features in indexing MWL variations across different tasks. The results showed that the amplitudes of tir-aERP components, N1, early P3a, late P3a, and the reorienting negativity, significantly decreased with the increasing MWL in both N-back and MATB. Task type had no obvious influence on the amplitudes and topological layout of the MWL-sensitive tir-aERP features. The relative PSDs in θ, α, and low β bands were also sensitive to MWL variations. However, the MWL-sensitive PSD features and their topological patterns were significantly affected by task type. The cross-task classification results based on tir-aERP features also significantly outperformed the PSD features. These results suggest that the tir-aERPs should be potentially more consistent MWL indicators across very different task types when compared to PSD. The current study may provide new insights to our understanding of the common and distinctive neuropsychological essences of MWL across different tasks.

Decoding different working memory states during an operation span task from prefrontal fNIRS signals

We propose an effective and practical decoding method of different mental states for potential applications for the design of brain-computer interfaces, prediction of cognitive behaviour, and investigation of cognitive mechanism. Functional near infrared spectroscopy (fNIRS) signals that interrogated the prefrontal and parietal cortices and were evaluated by generalized linear model were recorded when nineteen healthy adults performed the operation span (OSPAN) task. The oxygenated hemoglobin changes during OSPAN, response, and rest periods were classified with a support vector machine (SVM). The relevance vector regression algorithm was utilized for prediction of cognitive performance based on multidomain features of fNIRS signals from the OSPAN task. We acquired decent classification accuracies for OSPAN vs. response (above 91.2%) and for OSPAN vs. rest (above 94.7%). Eight of the ten cognitive testing scores could be predicted from the combination of OSPAN and response features, which indicated the brain hemodynamic responses contain meaningful information suitable for predicting cognitive performance.