Evaluation of an Adaptive Game that Uses EEG Measures Validated during the Design Process as Inputs to a Biocybernetic Loop

Burst c-VEP Based BCI: Optimizing stimulus design for enhanced classification with minimal calibration data and improved user experience

The utilization of aperiodic flickering visual stimuli under the form of code-modulated Visual Evoked Potentials (c-VEP) represents a pivotal advancement in the field of reactive Brain-Computer Interface (rBCI). A major advantage of the c-VEP approach is that the training of the model is independent of the number and complexity of targets, which helps reduce calibration time. Nevertheless, the existing designs of c-VEP stimuli can be further improved in terms of visual user experience but also to achieve a higher signal-to-noise ratio, while shortening the selection time and calibration process. In this study, we introduce an innovative variant of code-VEP, referred to as "Burst c-VEP". This original approach involves the presentation of short bursts of aperiodic visual flashes at a deliberately slow rate, typically ranging from two to four flashes per second. The rationale behind this design is to leverage the sensitivity of the primary visual cortex to transient changes in low-level stimuli features to reliably elicit distinctive series of visual evoked potentials. In comparison to other types of faster-paced code sequences, burst c-VEP exhibit favorable properties to achieve high bitwise decoding performance using convolutional neural networks (CNN), which yields potential to attain faster selection time with the need for less calibration data. Furthermore, our investigation focuses on reducing the perceptual saliency of c-VEP through the attenuation of visual stimuli contrast and intensity to significantly improve users' visual comfort. The proposed solutions were tested through an offline 4-classes c-VEP protocol involving 12 participants. Following a factorial design, participants were instructed to focus on c-VEP targets whose pattern (burst and maximum-length sequences) and amplitude (100% or 40% amplitude depth modulations) were manipulated across experimental conditions. Firstly, the full amplitude burst c-VEP sequences exhibited higher accuracy, ranging from 90.5% (with 17.6s of calibration data) to 95.6% (with 52.8s of calibration data), compared to its m-sequence counterpart (71.4% to 85.0%). The mean selection time for both types of codes (1.5 s) compared favorably to reports from previous studies. Secondly, our findings revealed that lowering the intensity of the stimuli only slightly decreased the accuracy of the burst code sequences to 94.2% while leading to substantial improvements in terms of user experience. Taken together, these results demonstrate the high potential of the proposed burst codes to advance reactive BCI both in terms of performance and usability. The collected dataset, along with the proposed CNN architecture implementation, are shared through open-access repositories.

Using machine learning methods and EEG to discriminate aircraft pilot cognitive workload during flight

Pilots of aircraft face varying degrees of cognitive workload even during normal flight operations. Periods of low cognitive workload may be followed by periods of high cognitive workload and vice versa. During such changing demands, there exists potential for increased error on behalf of the pilots due to periods of boredom or excessive cognitive task demand. To further understand cognitive workload in aviation, the present study involved collection of electroencephalogram (EEG) data from ten (10) collegiate aviation students in a live-flight environment in a single-engine aircraft. Each pilot possessed a Federal Aviation Administration (FAA) commercial pilot certificate and either FAA class I or class II medical certificate. Each pilot flew a standardized flight profile representing an average instrument flight training sequence. For data analysis, we used four main sub-bands of the recorded EEG signals: delta, theta, alpha, and beta. Power spectral density (PSD) and log energy entropy of each sub-band across 20 electrodes were computed and subjected to two feature selection algorithms (recursive feature elimination (RFE) and lasso cross-validation (LassoCV), and a stacking ensemble machine learning algorithm composed of support vector machine, random forest, and logistic regression. Also, hyperparameter optimization and tenfold cross-validation were used to improve the model performance, reliability, and generalization. The feature selection step resulted in 15 features that can be considered an indicator of pilots' cognitive workload states. Then these features were applied to the stacking ensemble algorithm, and the highest results were achieved using the selected features by the RFE algorithm with an accuracy of 91.67% (± 0.11), a precision of 93.89% (± 0.09), recall of 91.67% (± 0.11), F-score of 91.22% (± 0.12), and the mean ROC-AUC of 0.93 (± 0.06). The achieved results indicated that the combination of PSD and log energy entropy, along with well-designed machine learning algorithms, suggest the potential for the use of EEG to discriminate periods of the low, medium, and high workload to augment aircraft system design, including flight automation features to improve aviation safety.

Virtual Reality Adaptation Using Electrodermal Activity to Support the User Experience

Virtual reality is increasingly used for tasks such as work and education. Thus, rendering scenarios that do not interfere with such goals and deplete user experience are becoming progressively more relevant. We present a physiologically adaptive system that optimizes the virtual environment based on physiological arousal, i.e., electrodermal activity. We investigated the usability of the adaptive system in a simulated social virtual reality scenario. Participants completed an n-back task (primary) and a visual detection (secondary) task. Here, we adapted the visual complexity of the secondary task in the form of the number of non-player characters of the secondary task to accomplish the primary task. We show that an adaptive virtual reality can improve users’ comfort by adapting to physiological arousal regarding the task complexity. Our findings suggest that physiologically adaptive virtual reality systems can improve users’ experience in a wide range of scenarios.

Impact of Visual Game-Like Features on Cognitive Performance in a Virtual Reality Working Memory Task: Within-Subjects Experiment

BACKGROUND

Although the pursuit of improved cognitive function through working memory training has been the subject of decades of research, the recent growth in commercial adaptations of classic working memory tasks in the form of gamified apps warrants additional scrutiny. In particular, the emergence of virtual reality as a platform for cognitive training presents opportunities for the use of novel visual features.

OBJECTIVE

This study aimed to add to the body of knowledge regarding the use of game-like visual design elements by specifically examining the application of two particular visual features common to virtual reality environments: immersive, colorful backgrounds and the use of 3D depth. In addition, electroencephalography (EEG) data were collected to identify potential neural correlates of any observed changes in performance.

METHODS

A simple visual working memory task was presented to participants in several game-like adaptations, including the use of colorful, immersive backgrounds and 3D depth. The impact of each adaptation was separately assessed using both EEG and performance assessment outcomes and compared with an unmodified version of the task.

RESULTS

Results suggest that although accuracy and reaction time may be slightly affected by the introduction of such game elements, the effects were small and not statistically significant. Changes in EEG power, particularly in the beta and theta rhythms, were significant but failed to correlate with any corresponding changes in performance. Therefore, they may only reflect cognitive changes at the perceptual level.

CONCLUSIONS

Overall, the data suggest that the addition of these specific visual features to simple cognitive tasks does not appear to significantly affect performance or task-dependent cognitive load.

Dual Passive Reactive Brain-Computer Interface: A Novel Approach to Human-Machine Symbiosis

The present study proposes a novel concept of neuroadaptive technology, namely a dual passive-reactive Brain-Computer Interface (BCI), that enables bi-directional interaction between humans and machines. We have implemented such a system in a realistic flight simulator using the NextMind classification algorithms and framework to decode pilots' intention (reactive BCI) and to infer their level of attention (passive BCI). Twelve pilots used the reactive BCI to perform checklists along with an anti-collision radar monitoring task that was supervised by the passive BCI. The latter simulated an automatic avoidance maneuver when it detected that pilots missed an incoming collision. The reactive BCI reached 100% classification accuracy with a mean reaction time of 1.6 s when exclusively performing the checklist task. Accuracy was up to 98.5% with a mean reaction time of 2.5 s when pilots also had to fly the aircraft and monitor the anti-collision radar. The passive BCI achieved a F1-score of 0.94. This first demonstration shows the potential of a dual BCI to improve human-machine teaming which could be applied to a variety of applications.

Retrospective on the First Passive Brain-Computer Interface Competition on Cross-Session Workload Estimation

As is the case in several research domains, data sharing is still scarce in the field of Brain-Computer Interfaces (BCI), and particularly in that of passive BCIs-i.e., systems that enable implicit interaction or task adaptation based on a user's mental state(s) estimated from brain measures. Moreover, research in this field is currently hindered by a major challenge, which is tackling brain signal variability such as cross-session variability. Hence, with a view to develop good research practices in this field and to enable the whole community to join forces in working on cross-session estimation, we created the first passive brain-computer interface competition on cross-session workload estimation. This competition was part of the 3rd International Neuroergonomics conference. The data were electroencephalographic recordings acquired from 15 volunteers (6 females; average 25 y.o.) who performed 3 sessions-separated by 7 days-of the Multi-Attribute Task Battery-II (MATB-II) with 3 levels of difficulty per session (pseudo-randomized order). The data -training and testing sets-were made publicly available on Zenodo along with Matlab and Python toy code (https://doi.org/10.5281/zenodo.5055046). To this day, the database was downloaded more than 900 times (unique downloads of all version on the 10th of December 2021: 911). Eleven teams from 3 continents (31 participants) submitted their work. The best achieving processing pipelines included a Riemannian geometry-based method. Although better than the adjusted chance level (38% with an α at 0.05 for a 3-class classification problem), the results still remained under 60% of accuracy. These results clearly underline the real challenge that is cross-session estimation. Moreover, they confirmed once more the robustness and effectiveness of Riemannian methods for BCI. On the contrary, chance level results were obtained by one third of the methods-4 teams- based on Deep Learning. These methods have not demonstrated superior results in this contest compared to traditional methods, which may be due to severe overfitting. Yet this competition is the first step toward a joint effort to tackle BCI variability and to promote good research practices including reproducibility.

Making the Best Out of IT: Design and Development of Exergames for Older Adults With Mild Neurocognitive Disorder - A Methodological Paper

Background: Utilizing information technology (IT) systems, for example in form of computerized cognitive screening or exergame-based (also called active videogames) training, has gained growing interest for supporting healthy aging and to detect, prevent and treat neurocognitive disorders (NCD). To ameliorate the effectiveness of exergaming, the neurobiological mechanisms as well as the most effective components for exergame-based training remain to be established. At the same time, it is important to account for the end-users' capabilities, preferences, and therapeutic needs during the design and development process to foster the usability and acceptance of the resulting program in clinical practice. This will positively influence adherence to the resulting exergame-based training program, which, in turn, favors more distinct training-related neurobiological effects. Objectives and Methods: This methodological paper describes the design and development process of novel exergame-based training concepts guided by a recently proposed methodological framework: The 'Multidisciplinary Iterative Design of Exergames (MIDE): A Framework for Supporting the Design, Development, and Evaluation of Exergames for Health' (Li et al., 2020). Case Study: A step-by-step application of the MIDE-framework as a specific guidance in an ongoing project aiming to design, develop, and evaluate an exergame-based training concept with the aim to halt and/or reduce cognitive decline and improve quality of life in older adults with mild neurocognitive disorder (mNCD) is illustrated. Discussion and Conclusion: The development of novel exergame-based training concepts is greatly facilitated when it is based on a theoretical framework (e.g., the MIDE-framework). Applying this framework resulted in a structured, iterative, and evidence-based approach that led to the identification of multiple key requirements for the exergame design as well as the training components that otherwise may have been overlooked or neglected. This is expected to foster the usability and acceptance of the resulting exergame intervention in "real life" settings. Therefore, it is strongly recommended to implement a theoretical framework (e.g., the MIDE-framework) for future research projects in line with well-known checklists to improve completeness of reporting and replicability when serious games for motor-cognitive rehabilitation purposes are to be developed.

Classification of Game Demand and the Presence of Experimental Pain Using Functional Near-Infrared Spectroscopy

Pain tolerance can be increased by the introduction of an active distraction, such as a computer game. This effect has been found to be moderated by game demand, i.e., increased game demand = higher pain tolerance. A study was performed to classify the level of game demand and the presence of pain using implicit measures from functional Near-InfraRed Spectroscopy (fNIRS) and heart rate features from an electrocardiogram (ECG). Twenty participants played a racing game that was configured to induce low (Easy) or high (Hard) levels of demand. Both Easy and Hard levels of game demand were played with or without the presence of experimental pain using the cold pressor test protocol. Eight channels of fNIRS data were recorded from a montage of frontal and central-parietal sites located on the midline. Features were generated from these data, a subset of which were selected for classification using the RELIEFF method. Classifiers for game demand (Easy vs. Hard) and pain (pain vs. no-pain) were developed using five methods: Support Vector Machine (SVM), k-Nearest Neighbour (kNN), Naive Bayes (NB) and Random Forest (RF). These models were validated using a ten fold cross-validation procedure. The SVM approach using features derived from fNIRS was the only method that classified game demand at higher than chance levels (accuracy = 0.66, F1 = 0.68). It was not possible to classify pain vs. no-pain at higher than chance level. The results demonstrate the viability of utilising fNIRS data to classify levels of game demand and the difficulty of classifying pain when another task is present.

Enhancing Sustained Attention

Arguably, automation is fast transforming many enterprise business processes, transforming operational jobs into monitoring tasks. Consequently, the ability to sustain attention during extended periods of monitoring is becoming a critical skill. This manuscript presents a Brain-Computer Interface (BCI) prototype which seeks to combat decrements in sustained attention during monitoring tasks within an enterprise system. A brain-computer interface is a system which uses physiological signals output by the user as an input. The goal is to better understand human responses while performing tasks involving decision and monitoring cycles, finding ways to improve performance and decrease on-task error. Decision readiness and the ability to synthesize complex and abundant information in a brief period during critical events has never been more important. Closed-loop control and motivational control theory were synthesized to provide the basis from which a framework for a prototype was developed to demonstrate the feasibility and value of a BCI in critical enterprise activities. In this pilot study, the BCI was implemented and evaluated through laboratory experimentation using an ecologically valid task. The results show that the technological artifact allowed users to regulate sustained attention positively while performing the task. Levels of sustained attention were shown to be higher in the conditions assisted by the BCI. Furthermore, this increased cognitive response seems to be related to increased on-task action and a small reduction in on-task errors. The research concludes with a discussion of the future research directions and their application in the enterprise.

Impact of screen size on cognitive training task performance: An HMD study

To better understand the impact of different screen sizes in cognitive training, study subjects performed an adaptive training task at three separate visual angles using a head-mounted display (HMD). Cognitive load was assessed using EEG and compared with task performance (accuracy and response time) for each condition. While previous studies found performance benefits corresponding to increased screen size in memory and learning tasks, our results suggest such benefits may only apply up to a visual angle of approximately 20°, after which increases in size become inversely correlated with task performance.

User Experience With Dynamic Difficulty Adjustment Methods for an Affective Exergame: Comparative Laboratory-Based Study

BACKGROUND

In affective exergames, game difficulty is dynamically adjusted to match the user's physical and psychological state. Such an adjustment is commonly made based on a combination of performance measures (eg, in-game scores) and physiological measurements, which provide insight into the player's psychological state. However, although many prototypes of affective games have been presented and many studies have shown that physiological measurements allow more accurate classification of the player's psychological state than performance measures, few studies have examined whether dynamic difficulty adjustment (DDA) based on physiological measurements (which requires additional sensors) results in a better user experience than performance-based DDA or manual difficulty adjustment.

OBJECTIVE

This study aims to compare five DDA methods in an affective exergame: manual (player-controlled), random, performance-based, personality-performance-based, and physiology-personality-performance-based (all-data).

METHODS

A total of 50 participants (N=50) were divided into five groups, corresponding to the five DDA methods. They played an exergame version of Pong for 18 minutes, starting at a medium difficulty; every 2 minutes, two game difficulty parameters (ball speed and paddle size) were adjusted using the participant's assigned DDA method. The DDA rules for the performance-based, personality-performance-based, and all-data groups were developed based on data from a previous open-loop study. Seven physiological responses were recorded throughout the sessions, and participants self-reported their preferred changes to difficulty every 2 minutes. After playing the game, participants reported their in-game experience using two questionnaires: the Intrinsic Motivation Inventory and the Flow Experience Measure.

RESULTS

Although the all-data method resulted in the most accurate changes to ball speed and paddle size (defined as the percentage match between DDA choice and participants' preference), no significant differences between DDA methods were found on the Intrinsic Motivation Inventory and Flow Experience Measure. When the data from all four automated DDA methods were pooled together, the accuracy of changes in ball speed was significantly correlated with players' enjoyment (r=0.38) and pressure (r=0.43).

CONCLUSIONS

Although our study is limited by the use of a between-subjects design and may not generalize to other exergame designs, the results do not currently support the inclusion of physiological measurements in affective exergames, as they did not result in an improved user experience. As the accuracy of difficulty changes is correlated with user experience, the results support the development of more effective DDA methods. However, they show that the inclusion of physiological measurements does not guarantee a better user experience even if it yields promising results in offline cross-validation.

Forefront Users' Experience Evaluation by Employing Together Virtual Reality and Electroencephalography: A Case Study on Cognitive Effects of Scents

Scents have the ability to affect peoples' mental states and task performance with to different extents. It has been widely demonstrated that the lemon scent, included in most all-purpose cleaners, elicits stimulation and activation, while the lavender scent elicits relaxation and sedative effects. The present study aimed at investigating and fostering a novel approach to evaluate users' experience with respect to scents' effects through the joint employment of Virtual Reality and users' neurophysiological monitoring, in particular Electroencephalography. In particular, this study, involving 42 participants, aimed to compare the effects of lemon and lavender scents on the deployment of cognitive resources during a daily life experience consisting in a train journey carried out in virtual reality. Our findings showed a significant higher request of cognitive resources during the processing of an informative message for subjects exposed to the lavender scent with respect to the lemon exposure. No differences were found between lemon and lavender conditions on the self-reported items of pleasantness and involvement; as this study demonstrated, the employment of the lavender scent preserves the quality of the customer experience to the same extent as the more widely used lemon scent.

Using Fixation-Related Potentials for Inspecting Natural Interactions

Brain-Computer Interfaces (BCI) offer unique windows into the cognitive processes underlying human-machine interaction. Identifying and analyzing the appropriate brain activity to have access to such windows is often difficult due to technical or psycho-physiological constraints. Indeed, studying interactions through this approach frequently requires adapting them to accommodate specific BCI-related paradigms which change the functioning of their interface on both the human-side and the machine-side. The combined examination of Electroencephalography and Eyetracking recordings, mainly by means of studying Fixation-Related Potentials, can help to circumvent the necessity for these adaptations by determining interaction-relevant moments during natural manipulation. In this contribution, we examine how properties contained within the bi-modal recordings can be used to assess valuable information about the interaction. Practically, three properties are studied which can be obtained solely through data obtained from analysis of the recorded biosignals. Namely, these properties consist of relative gaze metrics, being abstractions of the gaze patterns, the amplitude variations in the early brain activity potentials and the brain activity frequency band differences between fixations. Through their observation, information about three different aspects of the explored interface are obtained. Respectively, the properties provide insights about general perceived task difficulty, locate moments of higher attentional effort and discriminate between moments of exploration and moments of active interaction.

Analysis of brain waves changes in stressful situations based on horror game with the implementation of virtual reality and brain-computer interface system: a case study

Objectives

This presents a case for fear and stress stimuli and afterward EEG data analysis.

Methods

The stress factor had been evoked by a computer horror game correlated with virtual reality (VR) and brain-computer interface (BCI) from OpenBCI, applied for the purpose of brain waves changes observation.

Results

Results obtained during the initial study were promising and provide conclusions for further research in this field carried out on an expanded group of involved participants.

Conclusions

The study provided very promising and interesting results. Further investigation with larger amount of participants will be carried out.

A Psychophysiological Model of Firearms Training in Police Officers: A Virtual Reality Experiment for Biocybernetic Adaptation

Crucial elements for police firearms training include mastering very specific psychophysiological responses associated with controlled breathing while shooting. Under high-stress situations, the shooter is affected by responses of the sympathetic nervous system that can impact respiration. This research focuses on how frontal oscillatory brainwaves and cardiovascular responses of trained police officers (N = 10) are affected during a virtual reality (VR) firearms training routine. We present data from an experimental study wherein shooters were interacting in a VR-based training simulator designed to elicit psychophysiological changes under easy, moderate and frustrating difficulties. Outcome measures in this experiment include electroencephalographic and heart rate variability (HRV) parameters, as well as performance metrics from the VR simulator. Results revealed that specific frontal areas of the brain elicited different responses during resting states when compared with active shooting in the VR simulator. Moreover, sympathetic signatures were found in the HRV parameters (both time and frequency) reflecting similar differences. Based on the experimental findings, we propose a psychophysiological model to aid the design of a biocybernetic adaptation layer that creates real-time modulations in simulation difficulty based on targeted physiological responses.

A Neuroergonomics Approach to Mental Workload, Engagement and Human Performance

The assessment and prediction of cognitive performance is a key issue for any discipline concerned with human operators in the context of safety-critical behavior. Most of the research has focused on the measurement of mental workload but this construct remains difficult to operationalize despite decades of research on the topic. Recent advances in Neuroergonomics have expanded our understanding of neurocognitive processes across different operational domains. We provide a framework to disentangle those neural mechanisms that underpin the relationship between task demand, arousal, mental workload and human performance. This approach advocates targeting those specific mental states that precede a reduction of performance efficacy. A number of undesirable neurocognitive states (mind wandering, effort withdrawal, perseveration, inattentional phenomena) are identified and mapped within a two-dimensional conceptual space encompassing task engagement and arousal. We argue that monitoring the prefrontal cortex and its deactivation can index a generic shift from a nominal operational state to an impaired one where performance is likely to degrade. Neurophysiological, physiological and behavioral markers that specifically account for these states are identified. We then propose a typology of neuroadaptive countermeasures to mitigate these undesirable mental states.

Estimating Cognitive Workload in an Interactive Virtual Reality Environment Using EEG

With the recent surge of affordable, high-performance virtual reality (VR) headsets, there is unlimited potential for applications ranging from education, to training, to entertainment, to fitness and beyond. As these interfaces continue to evolve, passive user-state monitoring can play a key role in expanding the immersive VR experience, and tracking activity for user well-being. By recording physiological signals such as the electroencephalogram (EEG) during use of a VR device, the user's interactions in the virtual environment could be adapted in real-time based on the user's cognitive state. Current VR headsets provide a logical, convenient, and unobtrusive framework for mounting EEG sensors. The present study evaluates the feasibility of passively monitoring cognitive workload via EEG while performing a classical n-back task in an interactive VR environment. Data were collected from 15 participants and the spatio-spectral EEG features were analyzed with respect to task performance. The results indicate that scalp measurements of electrical activity can effectively discriminate three workload levels, even after suppression of a co-varying high-frequency activity.

Toward a Hybrid Passive BCI for the Modulation of Sustained Attention Using EEG and fNIRS

We report results of a study that utilizes a BCI to drive an interactive interface countermeasure that allows users to self-regulate sustained attention while performing an ecologically valid, long-duration business logistics task. An engagement index derived from EEG signals was used to drive the BCI while fNIRS measured hemodynamic activity for the duration of the task. Participants (n = 30) were split into three groups (1) no countermeasures (NOCM), (2) continuous countermeasures (CCM), and (3) event synchronized, level-dependent countermeasures (ECM). We hypothesized that the ability to self-regulate sustained attention through a neurofeedback mechanism would result in greater task engagement, decreased error rate and improved task performance. Data were analyzed by wavelet coherence analysis, statistical analysis, performance metrics and self-assessed cognitive workload via RAW-TLX. We found that when the BCI was used to deliver continuous interface countermeasures (CCM), task performance was moderately enhanced in terms of total 14,785 (σ = 423) and estimated missed sales 7.46% (σ = 1.76) when compared to the NOCM 14,529 (σ = 510), 9.79% (σ = 2.75), and the ECM 14,180 (σ = 875), 9.62% (σ = 4.91) groups. An "actions per minute" (APM) metric was used to determine interface interaction activity which showed that overall the CCM and ECM groups had a higher APM of 3.460 (SE = 0.140) and 3.317 (SE = 0.139) respectively when compared with the NOCM group 2.65 (SE = 0.097). Statistical analysis showed a significant difference between ECM - NOCM and CCM - NOCM (p < 0.001) groups, but no significant difference between the ECM - CCM groups. Analysis of the RAW-TLX scores showed that the CCM group had lowest total score 7.27 (σ = 3.1) when compared with the ECM 9.7 (σ = 3.3) and NOCM 9.2 (σ = 3.4) groups. No statistical difference was found between the RAW-TLX or the subscales, except for self-perceived performance (p < 0.028) comparing the CCM and ECM groups. The results suggest that providing a means to self-regulate sustained attention has the potential to keep operators engaged over long periods, and moderately increase on-task performance while decreasing on-task error.

Monitoring Pilot's Mental Workload Using ERPs and Spectral Power with a Six-Dry-Electrode EEG System in Real Flight Conditions

Recent technological progress has allowed the development of low-cost and highly portable brain sensors such as pre-amplified dry-electrodes to measure cognitive activity out of the laboratory. This technology opens promising perspectives to monitor the "brain at work" in complex real-life situations such as while operating aircraft. However, there is a need to benchmark these sensors in real operational conditions. We therefore designed a scenario in which twenty-two pilots equipped with a six-dry-electrode EEG system had to perform one low load and one high load traffic pattern along with a passive auditory oddball. In the low load condition, the participants were monitoring the flight handled by a flight instructor, whereas they were flying the aircraft in the high load condition. At the group level, statistical analyses disclosed higher P300 amplitude for the auditory target (Pz, P4 and Oz electrodes) along with higher alpha band power (Pz electrode), and higher theta band power (Oz electrode) in the low load condition as compared to the high load one. Single trial classification accuracy using both event-related potentials and event-related frequency features at the same time did not exceed chance level to discriminate the two load conditions. However, when considering only the frequency features computed over the continuous signal, classification accuracy reached around 70% on average. This study demonstrates the potential of dry-EEG to monitor cognition in a highly ecological and noisy environment, but also reveals that hardware improvement is still needed before it can be used for everyday flight operations.

Error-related negativity predicts failure in competitive dual-player video games

Along with improvement in electroencephalogram (EEG)-measurement technology, limitations on the situations in which data can be recorded are gradually being overcome. EEG measurement in real environments has become increasingly important as a means to monitor brain activity in our daily lives, such as while playing consumer games in the living room. The present study measured brain EEG activity while two players engaged in a competitive consumer baseball game in conditions that closely resembled daily life. The recorded brain activity was thus likely related to natural mental reactions and cognitive function that occur in similar daily life activities. To measure the EEG from participants who freely moved while playing the game, we developed EEG devices that incorporated a wireless time synchronization system using Global Positioning Satellite (GPS) signals. These devices stamped the time obtained from the GPS signals onto each data sample, which was then used to synchronize the data that were recorded by different devices. When the batter in the game swung and missed, the error-related negativity component of the event-related EEG potential was strongly evoked in frontal electrodes of the participant controlling the batter. Furthermore, the error-related negativity was modulated according to who was winning and by how much. Thus, here we have demonstrated "real-world" brain activity using a competitive consumer game, which increases intrinsic participant motivation.

Dynamic Threshold Selection for a Biocybernetic Loop in an Adaptive Video Game Context

Passive Brain-Computer interfaces (pBCIs) are a human-computer communication tool where the computer can detect from neurophysiological signals the current mental or emotional state of the user. The system can then adjust itself to guide the user toward a desired state. One challenge facing developers of pBCIs is that the system's parameters are generally set at the onset of the interaction and remain stable throughout, not adapting to potential changes over time such as fatigue. The goal of this paper is to investigate the improvement of pBCIs with settings adjusted according to the information provided by a second neurophysiological signal. With the use of a second signal, making the system a hybrid pBCI, those parameters can be continuously adjusted with dynamic thresholding to respond to variations such as fatigue or learning. In this experiment, we hypothesize that the adaptive system with dynamic thresholding will improve perceived game experience and objective game performance compared to two other conditions: an adaptive system with single primary signal biocybernetic loop and a control non-adaptive game. A within-subject experiment was conducted with 16 participants using three versions of the game Tetris. Each participant plays 15 min of Tetris under three experimental conditions. The control condition is the traditional game of Tetris with a progressive increase in speed. The second condition is a cognitive load only biocybernetic loop with the parameters presented in Ewing et al. (2016). The third condition is our proposed biocybernetic loop using dynamic threshold selection. Electroencephalography was used as the primary signal and automatic facial expression analysis as the secondary signal. Our results show that, contrary to our expectations, the adaptive systems did not improve the participants' experience as participants had more negative affect from the BCI conditions than in the control condition. We endeavored to develop a system that improved upon the authentic version of the Tetris game, however, our proposed adaptive system neither improved players' perceived experience, nor their objective performance. Nevertheless, this experience can inform developers of hybrid passive BCIs on a novel way to employ various neurophysiological features simultaneously.