Biocybernetic system evaluates indices of operator engagement in automated task

Mild Normobaric Hypoxia Exposure for Human-Autonomy System Testing

An experiment investigated the impact of normobaric hypoxia induction on aircraft pilot performance to specifically evaluate the use of hypoxia as a method to induce mild cognitive impairment to explore human-autonomous systems integration opportunities. Results of this exploratory study show that the effect of 15,000 feet simulated altitude did not induce cognitive deficits as indicated by performance on written, computer-based, or simulated flight tasks. However, the subjective data demonstrated increased effort by the human test subject pilots to maintain equivalent performance in a flight simulation task. This study represents current research intended to add to the current knowledge of performance decrement and pilot workload assessment to improve automation support and increase aviation safety.

Improving Challenge/Skill Ratio in a Multimodal Interface by Simultaneously Adapting Game Difficulty and Haptic Assistance through Psychophysiological and Performance Feedback

In order to harmonize robotic devices with human beings, the robots should be able to perceive important psychosomatic impact triggered by emotional states such as frustration or boredom. This paper presents a new type of biocooperative control architecture, which acts toward improving the challenge/skill relation perceived by the user when interacting with a robotic multimodal interface in a cooperative scenario. In the first part of the paper, open-loop experiments revealed which physiological signals were optimal for inclusion in the feedback loop. These were heart rate, skin conductance level, and skin conductance response frequency. In the second part of the paper, the proposed controller, consisting of a biocooperative architecture with two degrees of freedom, simultaneously modulating game difficulty and haptic assistance through performance and psychophysiological feedback, is presented. With this setup, the perceived challenge can be modulated by means of the game difficulty and the perceived skill by means of the haptic assistance. A new metric (FlowIndex) is proposed to numerically quantify and visualize the challenge/skill relation. The results are contrasted with comparable previously published work and show that the new method afforded a higher FlowIndex (i.e., a superior challenge/skill relation) and an improved balance between augmented performance and user satisfaction (higher level of valence, i.e., a more enjoyable and satisfactory experience).

Human Factors and Neurophysiological Metrics in Air Traffic Control: A Critical Review

This paper provides a focused and organized review of the research progress on neurophysiological indicators, also called "neurometrics," to show how they can effectively address some of the most important human factors (HFs) needs in the air traffic management (ATM) field. In order to better understand and highlight available opportunities of such neuroscientific applications, state of the art on the most involved HFs and related cognitive processes (e.g., mental workload and cognitive training) are presented together with examples of possible applications in current and future ATM scenarios. Furthermore, this paper will discuss the potential enhancements that further research and development activities could bring to the efficiency and safety of the ATM service.

The knowns and unknowns of boredom: a review of the literature

Despite the ubiquitous nature of boredom, the definition, function, and correlates of boredom are still poorly understood. In this review, we summarize the "known" (consistent evidence) and "unknown" (inconsistent evidence) correlates of boredom. We show that boredom is consistently related to negative affect, task-unrelated thought, over-estimation of elapsed time, reduced agency, as well as to over- and under-stimulation. Activation of the default mode network was consistent across the few available fMRI studies, while the recruitment of other brain areas such as the hippocampus and anterior insular cortex, was a notable but less consistent correlate of boredom. Other less consistent correlates of boredom are also reviewed, such as the level of arousal and the mental attributions given to fluctuations of attention. Finally, we identify two critical factors that may contribute to current inconsistencies in the literature and may hamper further progress in the field. First, there is relatively little consistency in the way in which boredom has been operationalized across studies to date, with operationalizations of boredom ranging from negative affect paired with under-stimulation, over-stimulation, to negative affect paired with a lack of goal-directed actions. Second, preliminary evidence suggests the existence of distinct types of boredom (e.g., searching vs. apathetic) that may have different and sometimes even opposing correlates. Adopting a more precise and consistent way of operationalizing boredom, and arriving at an empirically validated taxonomy of different types of boredom, could serve to overcome the current roadblocks to facilitate further progress in our scientific understanding of boredom.

Design of an Adaptive Human-Machine System Based on Dynamical Pattern Recognition of Cognitive Task-Load

This paper developed a cognitive task-load (CTL) classification algorithm and allocation strategy to sustain the optimal operator CTL levels over time in safety-critical human-machine integrated systems. An adaptive human-machine system is designed based on a non-linear dynamic CTL classifier, which maps a set of electroencephalogram (EEG) and electrocardiogram (ECG) related features to a few CTL classes. The least-squares support vector machine (LSSVM) is used as dynamic pattern classifier. A series of electrophysiological and performance data acquisition experiments were performed on seven volunteer participants under a simulated process control task environment. The participant-specific dynamic LSSVM model is constructed to classify the instantaneous CTL into five classes at each time instant. The initial feature set, comprising 56 EEG and ECG related features, is reduced to a set of 12 salient features (including 11 EEG-related features) by using the locality preserving projection (LPP) technique. An overall correct classification rate of about 80% is achieved for the 5-class CTL classification problem. Then the predicted CTL is used to adaptively allocate the number of process control tasks between operator and computer-based controller. Simulation results showed that the overall performance of the human-machine system can be improved by using the adaptive automation strategy proposed.

Evaluation of a Dry EEG System for Application of Passive Brain-Computer Interfaces in Autonomous Driving

We tested the applicability and signal quality of a 16 channel dry electroencephalography (EEG) system in a laboratory environment and in a car under controlled, realistic conditions. The aim of our investigation was an estimation how well a passive Brain-Computer Interface (pBCI) can work in an autonomous driving scenario. The evaluation considered speed and accuracy of self-applicability by an untrained person, quality of recorded EEG data, shifts of electrode positions on the head after driving-related movements, usability, and complexity of the system as such and wearing comfort over time. An experiment was conducted inside and outside of a stationary vehicle with running engine, air-conditioning, and muted radio. Signal quality was sufficient for standard EEG analysis in the time and frequency domain as well as for the use in pBCIs. While the influence of vehicle-induced interferences to data quality was insignificant, driving-related movements led to strong shifts in electrode positions. In general, the EEG system used allowed for a fast self-applicability of cap and electrodes. The assessed usability of the system was still acceptable while the wearing comfort decreased strongly over time due to friction and pressure to the head. From these results we conclude that the evaluated system should provide the essential requirements for an application in an autonomous driving context. Nevertheless, further refinement is suggested to reduce shifts of the system due to body movements and increase the headset's usability and wearing comfort.

The Berlin Brain-Computer Interface: Progress Beyond Communication and Control

The combined effect of fundamental results about neurocognitive processes and advancements in decoding mental states from ongoing brain signals has brought forth a whole range of potential neurotechnological applications. In this article, we review our developments in this area and put them into perspective. These examples cover a wide range of maturity levels with respect to their applicability. While we assume we are still a long way away from integrating Brain-Computer Interface (BCI) technology in general interaction with computers, or from implementing neurotechnological measures in safety-critical workplaces, results have already now been obtained involving a BCI as research tool. In this article, we discuss the reasons why, in some of the prospective application domains, considerable effort is still required to make the systems ready to deal with the full complexity of the real world.

Closing the Loop of an Adaptive System with Cognitive State

This paper describes an adaptive system that “closes the loop” by utilizing a real-time, directly sensed measure of cognitive state of the human operator. The Honeywell Augmented Cognition team has developed a Closed Loop Integrated Prototype (CLIP) of a Communications Scheduler, for application to the U.S. Army's Future Force Warrior (FFW) program. It is expected that in a highly networked environment the sheer magnitude of communication traffic could overwhelm the individual soldier. The CLIP exploits real-time neurophysiological and physiological measurements of the human operator in order to create a cognitive state profile, which is used to augment the work environment to improve human-automation joint performance. An experiment showed that the Communications Scheduler enabled higher situation awareness and message comprehension in high workload conditions. Based solely on cognitive state, the system inferred a subject's message comprehension and repeated unattended messages in the majority of cases, without yielding an unacceptably high false alarm rate.

Performance Enhancement with Real-Time Physiologically Controlled Adaptive Aiding

The realization of optimal system performance is the goal of both system designers and users. One critical component in attaining this goal is proper operator functioning. In contemporary systems the functional state of the operator is not considered during system operation. Degraded states of operator functioning can result from the demands of controlling complex systems, the work environment and internal operator variables. This, in turn, can lead to errors and overall suboptimal system performance. In the case of mental workload, system performance could be improved by reducing task demands during periods of operator overload. Accurate estimation of the operator's functional state is crucial to successful implementation of an adaptive aiding system. One method of determining operator functional state is by monitoring the operator's physiology. In the present study, physiological signals were used to continuously monitor subject's functional state and to adapt the task by reducing the number of subtasks when high levels of mental workload were detected. The goal was to demonstrate performance improvement with adaptive aiding. Because adaptive aiding during high mental workload has not been previously implemented its benefit has not be demonstrated. Application of adaptive aiding techniques reduced tracking task error by 44% and resource monitoring error by 33%. These results demonstrate the utility of adaptive aiding using physiological measures with artificial neural networks to determine the appropriate time to introduce the aiding.

Adaptive Automation Triggered by EEG-Based Mental Workload Index: A Passive Brain-Computer Interface Application in Realistic Air Traffic Control Environment

Adaptive Automation (AA) is a promising approach to keep the task workload demand within appropriate levels in order to avoid both the under- and over-load conditions, hence enhancing the overall performance and safety of the human-machine system. The main issue on the use of AA is how to trigger the AA solutions without affecting the operative task. In this regard, passive Brain-Computer Interface (pBCI) systems are a good candidate to activate automation, since they are able to gather information about the covert behavior (e.g., mental workload) of a subject by analyzing its neurophysiological signals (i.e., brain activity), and without interfering with the ongoing operational activity. We proposed a pBCI system able to trigger AA solutions integrated in a realistic Air Traffic Management (ATM) research simulator developed and hosted at ENAC (École Nationale de l'Aviation Civile of Toulouse, France). Twelve Air Traffic Controller (ATCO) students have been involved in the experiment and they have been asked to perform ATM scenarios with and without the support of the AA solutions. Results demonstrated the effectiveness of the proposed pBCI system, since it enabled the AA mostly during the high-demanding conditions (i.e., overload situations) inducing a reduction of the mental workload under which the ATCOs were operating. On the contrary, as desired, the AA was not activated when workload level was under the threshold, to prevent too low demanding conditions that could bring the operator's workload level toward potentially dangerous conditions of underload.

Measuring Implicit Cognitive and Emotional Engagement to Better Understand Learners’ Performance in Problem Solving

Abstract. Novel data collection methods and analysis algorithms developed in the field of neuroergonomics have opened new possibilities for research in education. Psychophysiological data can characterize the cognitive and emotional dimensions of engagement. This paper aims to describe the application of this research methodology to synchronously measure emotional and cognitive engagement during learning tasks.

Implementation and Validation of Engagement Monitoring in an Engagement Enhancing Rehabilitation System

Enhancing engagement of patients during stroke rehabilitation exercises are in the focus of current research. Various methods and computer supported tools have been developed for this purpose, which try to avoid mundane exercising that is prone to become a routine or even boring for the patients and leads to ineffective training. This paper introduces an engagement enhancing cyber-physical stroke rehabilitation system (CP-SRS) aiming at enhancing the patient's engagement during rehabilitation training exercises. This paper focuses on introducing the implementation and validation of the engagement monitoring subsystem (EMS) in the CP-SRS. The EMS is expected to evaluate the patient's actual engagement levels in motor, perceptive, cognitive and emotional aspects. Experiments in these four aspects were conducted separately, in order to characterize the range and accuracy of the engagement indicators by influencing the subjects into different engaged states. During the experiments, different setups were created to mimic the situations in which the subject was engaged or not engaged. The subjects involved in the experiments were healthy subjects. Results showed that the measurement in motor, perceptive, cognitive, and emotional aspects can represent the corresponding engagement level. More experiments will be conducted in the future to validate the efficiency of the CP-SRS in enhancing the engagement with stroke patients.

An Intelligent Man-Machine Interface-Multi-Robot Control Adapted for Task Engagement Based on Single-Trial Detectability of P300

Advanced man-machine interfaces (MMIs) are being developed for teleoperating robots at remote and hardly accessible places. Such MMIs make use of a virtual environment and can therefore make the operator immerse him-/herself into the environment of the robot. In this paper, we present our developed MMI for multi-robot control. Our MMI can adapt to changes in task load and task engagement online. Applying our approach of embedded Brain Reading we improve user support and efficiency of interaction. The level of task engagement was inferred from the single-trial detectability of P300-related brain activity that was naturally evoked during interaction. With our approach no secondary task is needed to measure task load. It is based on research results on the single-stimulus paradigm, distribution of brain resources and its effect on the P300 event-related component. It further considers effects of the modulation caused by a delayed reaction time on the P300 component evoked by complex responses to task-relevant messages. We prove our concept using single-trial based machine learning analysis, analysis of averaged event-related potentials and behavioral analysis. As main results we show (1) a significant improvement of runtime needed to perform the interaction tasks compared to a setting in which all subjects could easily perform the tasks. We show that (2) the single-trial detectability of the event-related potential P300 can be used to measure the changes in task load and task engagement during complex interaction while also being sensitive to the level of experience of the operator and (3) can be used to adapt the MMI individually to the different needs of users without increasing total workload. Our online adaptation of the proposed MMI is based on a continuous supervision of the operator's cognitive resources by means of embedded Brain Reading. Operators with different qualifications or capabilities receive only as many tasks as they can perform to avoid mental overload as well as mental underload.

Evaluation of an Adaptive Game that Uses EEG Measures Validated during the Design Process as Inputs to a Biocybernetic Loop

Biocybernetic adaptation is a form of physiological computing whereby real-time data streaming from the brain and body is used by a negative control loop to adapt the user interface. This article describes the development of an adaptive game system that is designed to maximize player engagement by utilizing changes in real-time electroencephalography (EEG) to adjust the level of game demand. The research consists of four main stages: (1) the development of a conceptual framework upon which to model the interaction between person and system; (2) the validation of the psychophysiological inference underpinning the loop; (3) the construction of a working prototype; and (4) an evaluation of the adaptive game. Two studies are reported. The first demonstrates the sensitivity of EEG power in the (frontal) theta and (parietal) alpha bands to changing levels of game demand. These variables were then reformulated within the working biocybernetic control loop designed to maximize player engagement. The second study evaluated the performance of an adaptive game of Tetris with respect to system behavior and user experience. Important issues for the design and evaluation of closed-loop interfaces are discussed.

Dynamic filtering improves attentional state prediction with fNIRS

Brain activity can predict a person's level of engagement in an attentional task. However, estimates of brain activity are often confounded by measurement artifacts and systemic physiological noise. The optimal method for filtering this noise - thereby increasing such state prediction accuracy - remains unclear. To investigate this, we asked study participants to perform an attentional task while we monitored their brain activity with functional near infrared spectroscopy (fNIRS). We observed higher state prediction accuracy when noise in the fNIRS hemoglobin [Hb] signals was filtered with a non-stationary (adaptive) model as compared to static regression (84% ± 6% versus 72% ± 15%).

Physiological Modalities for Relaxation Skill Transfer in Biofeedback Games

We present an adaptive biofeedback game for teaching self-regulation of stress. Our approach consists of monitoring the user's physiology during gameplay and adapting the game using a positive feedback loop that rewards relaxing behaviors and penalizes states of high arousal. We evaluate the approach using a casual game under three biofeedback modalities: electrodermal activity, heart rate variability, and breathing rate. The three biosignals can be measured noninvasively with wearable sensors, and represent different degrees of voluntary control and selectivity toward arousal. We conducted an experiment trial with 25 participants to compare the three modalities against a standard treatment (deep breathing) and a control condition (the game without biofeedback). Our results indicate that breathing-based game biofeedback is more effective in inducing relaxation during treatment than the other four groups. Participants in this group also showed greater retention of the relaxation skills (without biofeedback) during a subsequent stressor.

EEG-based index for engagement level monitoring during sustained attention

This paper investigates the relation between mental engagement level and sustained attention in 9 healthy adults performing a Conners' "not-X" continuous performance test (CPT), while their electroencephalographic (EEG) activity was simultaneously acquired. Spectral powers were estimated and extracted in the classical EEG frequency bands. The engagement index (β/α) was calculated employing four different cortical montages suggested by the literature. Results show the efficacy of the estimated measures in detecting changes in mental state and its correlation with subject reaction times throughout the test. Moreover, the influence of the recording sites was proved underling the role of frontal cortex in maintaining a constant sustained attention level.

Assessing the Multiple Dimensions of Engagement to Characterize Learning: A Neurophysiological Perspective

In a recent theoretical synthesis on the concept of engagement, Fredricks, Blumenfeld and Paris defined engagement by its multiple dimensions: behavioral, emotional and cognitive. They observed that individual types of engagement had not been studied in conjunction, and little information was available about interactions or synergy between the dimensions; consequently, more studies would contribute to creating finely tuned teaching interventions. Benefiting from the recent technological advances in neurosciences, this paper presents a recently developed methodology to gather and synchronize data on multidimensional engagement during learning tasks. The technique involves the collection of (a) electroencephalography, (b) electrodermal, (c) eye-tracking, and (d) facial emotion recognition data on four different computers. This led to synchronization issues for data collected from multiple sources. Post synchronization in specialized integration software gives researchers a better understanding of the dynamics between the multiple dimensions of engagement. For curriculum developers, these data could provide informed guidelines for achieving better instruction/learning efficiency. This technique also opens up possibilities in the field of brain-computer interactions, where adaptive learning or assessment environments could be developed.

GASICA: generic automated stress induction and control application design of an application for controlling the stress state

In a multitude of research and therapy paradigms it is relevant to know, and desirably to control, the stress state of a patient or participant. Examples include research paradigms in which the stress state is the dependent or independent variable, or therapy paradigms where this state indicates the boundaries of the therapy. To our knowledge, no application currently exists that focuses specifically on the automated control of the stress state while at the same time being generic enough to be used in various therapy and research purposes. Therefore, we introduce GASICA, an application aimed at the automated control of the stress state in a multitude of therapy and research paradigms. The application consists of three components: a digital stressor game, a set of measurement devices, and a feedback model. These three components form a closed loop (called a biocybernetic loop by Pope et al. (1995) and Fairclough (2009) that continuously presents an acute psychological stressor, measures several physiological responses to this stressor, and adjusts the stressor intensity based on these measurements by means of the feedback model, hereby aiming to control the stress state. In this manner GASICA presents multidimensional and ecological valid stressors, whilst continuously in control of the form and intensity of the presented stressors, aiming at the automated control of the stress state. Furthermore, the application is designed as a modular open-source application to easily implement different therapy and research tasks using a high-level programming interface and configuration file, and allows for the addition of (existing) measurement equipment, making it usable for various paradigms.

Cardiovascular state changes in simulated work environments

The usefulness of cardiovascular measures as indicators of changes in cognitive workload has been addressed in several studies. In this paper the question is explored whether cardiovascular patterns in heart rate, blood pressure, baroreflex sensitivity and HRV that are found are consistent within and between two simulated working environments. Two studies, were performed, both with 21 participants: one in an ambulance dispatch simulation and one in a driving simulator. In the ambulance dispatcher task an initial strong increase in blood pressure is followed by a moderate on-going increase in blood pressure during the next hour of task performance. This pattern is accompanied by a strong increase in baroreflex sensitivity while heart rate decreases. In the driving simulator study, blood pressure initially increases but decreases almost to baseline level in the next hour. This pattern is accompanied by a decrease in baroreflex sensitivity, while heart rate decreases. Results of both studies are interpreted in terms of autonomic control (related to both sympathetic and para-sympathetic effects), using a simplified simulation of a baroreflex regulation model. Interpretation of the results leads to the conclusion that the cardiovascular response patterns in both tasks are a combination of an initial defensive reaction, in combination with compensatory blood pressure control. The level of compensatory blood pressure control, however, is quite different for the two tasks. This helps to understand the differences in response patterns between the two studies in this paper and may be helpful as well for understanding differences in cardiovascular response patterns in general. A substantial part of the effects observed during task performance are regulatory effects and are not always directly related to workload manipulations. Making this distinction may also contribute to the understanding of differences in cardiovascular response patterns during cognitive workload.

Fast mental states decoding in mixed reality

The combination of Brain-Computer Interface (BCI) technology, allowing online monitoring and decoding of brain activity, with virtual and mixed reality (MR) systems may help to shape and guide implicit and explicit learning using ecological scenarios. Real-time information of ongoing brain states acquired through BCI might be exploited for controlling data presentation in virtual environments. Brain states discrimination during mixed reality experience is thus critical for adapting specific data features to contingent brain activity. In this study we recorded electroencephalographic (EEG) data while participants experienced MR scenarios implemented through the eXperience Induction Machine (XIM). The XIM is a novel framework modeling the integration of a sensing system that evaluates and measures physiological and psychological states with a number of actuators and effectors that coherently reacts to the user's actions. We then assessed continuous EEG-based discrimination of spatial navigation, reading and calculation performed in MR, using linear discriminant analysis (LDA) and support vector machine (SVM) classifiers. Dynamic single trial classification showed high accuracy of LDA and SVM classifiers in detecting multiple brain states as well as in differentiating between high and low mental workload, using a 5 s time-window shifting every 200 ms. Our results indicate overall better performance of LDA with respect to SVM and suggest applicability of our approach in a BCI-controlled MR scenario. Ultimately, successful prediction of brain states might be used to drive adaptation of data representation in order to boost information processing in MR.

Augmented Cognition using Real-time EEG-based Adaptive Strategies for Air Traffic Control

Future air traffic systems aim at increasing both the capacity and safety of the system, necessitating the development of new metrics and advisory tools for controllers’ workload in real-time. Psychophysiologi-cal data such as Electroencephalography (EEG) are used to contrast and validate subjective assessments and workload indices. EEG used within augmented cognition systems form situation awareness advisory tools that are able to provide real-time feedback to air-traffic control supervisors and planners. This aug-mented cognition system and experiments using the system with air traffic controllers are presented. Traf-fic indicators are used in conjunction with EEG-driven cognitive indicators to adapt the traffic in real-time through Computational Red Teaming (CRT) based adaptive control mechanisms. The metrics, measures, and adaptive control mechanisms are described and evaluated. The best mechanism to improve system ef-ficacy was found when the system allowed for real-time adaptation of traffic based on engagement met-rics driven from the EEG data.

Cognitive State Assessment

This effort investigated the ability of a neurophysiological measure to detect changes in workload during a task which is sensitive to cognitive function. A growing collection of research suggests that physiological measures such as EEG can be used to inform the adaptation of systems. However, it has been proposed that such measures often provide a gross interpretation of cognitive workload during complex tasks and are not sensitive to differences in specific cognitive function. To understand the utility of neurophysiological measures for human-machine interaction, we must know if these measures are sensitive to tasks which are sensitive to changes in cognitive function. To begin to answer this question, we investigated the sensitivity of Advanced Brain Monitoring’s EEG-based measures to changes in workload experienced during a Stroop task. Results indicated that ABM’s workload measure can detect changes associated with the attentional demands and cognitive processes linked to the ability to inhibit word naming during tasks involving semantic interference. This indicates that changes in workload associated with the ability to inhibit competing cognitive processes can be identified using neurophysiological workload measures.

Use of EEG workload indices for diagnostic monitoring of vigilance decrement

OBJECTIVE

A study was run to test which of five electroencephalographic (EEG) indices was most diagnostic of loss of vigilance at two levels of workload.

BACKGROUND

EEG indices of alertness include conventional spectral power measures as well as indices combining measures from multiple frequency bands, such as the Task Load Index (TLI) and the Engagement Index (El). However, it is unclear which indices are optimal for early detection of loss of vigilance.

METHOD

Ninety-two participants were assigned to one of two experimental conditions, cued (lower workload) and uncued (higher workload), and then performed a 40-min visual vigilance task. Performance on this task is believed to be limited by attentional resource availability. EEG was recorded continuously. Performance, subjective state, and workload were also assessed.

RESULTS

The task showed a vigilance decrement in performance; cuing improved performance and reduced subjective workload. Lower-frequency alpha (8 to 10.9 Hz) and TLI were most sensitive to the task parameters. The magnitude of temporal change was larger for lower-frequency alpha. Surprisingly, higher TLI was associated with superior performance. Frontal theta and El were influenced by task workload only in the final period of work. Correlational data also suggested that the indices are distinct from one another.

CONCLUSIONS

Lower-frequency alpha appears to be the optimal index for monitoring vigilance on the task used here, but further work is needed to test how diagnosticity of EEG indices varies with task demands.

APPLICATION

Lower-frequency alpha may be used to diagnose loss of operator alertness on tasks requiring vigilance.

Identification of temporal variations in mental workload using locally-linear-embedding-based EEG feature reduction and support-vector-machine-based clustering and classification techniques

Identifying the abnormal changes of mental workload (MWL) over time is quite crucial for preventing the accidents due to cognitive overload and inattention of human operators in safety-critical human-machine systems. It is known that various neuroimaging technologies can be used to identify the MWL variations. In order to classify MWL into a few discrete levels using representative MWL indicators and small-sized training samples, a novel EEG-based approach by combining locally linear embedding (LLE), support vector clustering (SVC) and support vector data description (SVDD) techniques is proposed and evaluated by using the experimentally measured data. The MWL indicators from different cortical regions are first elicited by using the LLE technique. Then, the SVC approach is used to find the clusters of these MWL indicators and thereby to detect MWL variations. It is shown that the clusters can be interpreted as the binary class MWL. Furthermore, a trained binary SVDD classifier is shown to be capable of detecting slight variations of those indicators. By combining the two schemes, a SVC-SVDD framework is proposed, where the clear-cut (smaller) cluster is detected by SVC first and then a subsequent SVDD model is utilized to divide the overlapped (larger) cluster into two classes. Finally, three-class MWL levels (low, normal and high) can be identified automatically. The experimental data analysis results are compared with those of several existing methods. It has been demonstrated that the proposed framework can lead to acceptable computational accuracy and has the advantages of both unsupervised and supervised training strategies.

A strategically timed verbal task improves performance and neurophysiological alertness during fatiguing drives

OBJECTIVE

The objective of this study was to investigate if a verbal task can improve alertness and if performance changes are associated with changes in alertness as measured by EEG.

BACKGROUND

Previous research has shown that a secondary task can improve performance on a short, monotonous drive. The current work extends this by examining longer, fatiguing drives. The study also uses EEG to confirm that improved driving performance is concurrent with improved driver alertness.

METHOD

A 90-min, monotonous simulator drive was used to place drivers in a fatigued state. Four secondary tasks were used: no verbal task, continuous verbal task, late verbal task, and a passive radio task.

RESULTS

When engaged in a secondary verbal task at the end of the drive, drivers showed improved lane-keeping performance and had improvements in neurophysiological measures of alertness.

CONCLUSION

A strategically timed concurrent task can improve performance even for fatiguing drives.

APPLICATION

Secondary-task countermeasures may prove useful for enhancing driving performance across a range of driving conditions.

Classification of intelligence quotient via brainwave sub-band power ratio features and artificial neural network

This paper elaborates on the novel intelligence assessment method using the brainwave sub-band power ratio features. The study focuses only on the left hemisphere brainwave in its relaxed state. Distinct intelligence quotient groups have been established earlier from the score of the Raven Progressive Matrices. Sub-band power ratios are calculated from energy spectral density of theta, alpha and beta frequency bands. Synthetic data have been generated to increase dataset from 50 to 120. The features are used as input to the artificial neural network. Subsequently, the brain behaviour model has been developed using an artificial neural network that is trained with optimized learning rate, momentum constant and hidden nodes. Findings indicate that the distinct intelligence quotient groups can be classified from the brainwave sub-band power ratios with 100% training and 88.89% testing accuracies.

A Hierarchical Probabilistic Framework for Recognizing Learners’ Interaction Experience Trends and Emotions

We seek to model the users’ experience within an interactive learning environment. More precisely, we are interested in assessing the relationship between learners’ emotional reactions and three trends in the interaction experience, namely,flow: the optimal interaction (a perfect immersion within the task),stuck: the nonoptimal interaction (a difficulty to maintain focused attention), andoff-task: the noninteraction (a dropout from the task). We propose a hierarchical probabilistic framework using a dynamic Bayesian network to model this relationship and to simultaneously recognize the probability of experiencing each trend as well as the emotional responses occurring subsequently. The framework combines three modalitydiagnostic variablesthat sense the learner’s experience including physiology, behavior, and performance,predictive variablesthat represent the current context and the learner’s profile, and adynamic structurethat tracks the evolution of the learner’s experience. An experimental study, with a specifically designed protocol for eliciting the targeted experiences, was conducted to validate our approach. Results revealed that multiple concurrent emotions can be associated with the experiences of flow, stuck, and off-task and that the same trend can be expressed differently from one individual to another. The evaluation of the framework showed promising results in predicting learners’ experience trends and emotional responses.

Monitoring attentional state with fNIRS

The ability to distinguish between high and low levels of task engagement in the real world is important for detecting and preventing performance decrements during safety-critical operational tasks. We therefore investigated whether functional Near Infrared Spectroscopy (fNIRS), a portable brain neuroimaging technique, can be used to distinguish between high and low levels of task engagement during the performance of a selective attention task. A group of participants performed the multi-source interference task (MSIT) while we recorded brain activity with fNIRS from two brain regions. One was a key region of the “task-positive” network, which is associated with relatively high levels of task engagement. The second was a key region of the “task-negative” network, which is associated with relatively low levels of task engagement (e.g., resting and not performing a task). Using activity in these regions as inputs to a multivariate pattern classifier, we were able to predict above chance levels whether participants were engaged in performing the MSIT or resting. We were also able to replicate prior findings from functional magnetic resonance imaging (fMRI) indicating that activity in task-positive and task-negative regions is negatively correlated during task performance. Finally, data from a companion fMRI study verified our assumptions about the sources of brain activity in the fNIRS experiment and established an upper bound on classification accuracy in our task. Together, our findings suggest that fNIRS could prove quite useful for monitoring cognitive state in real-world settings.

Classifying human operator functional state based on electrophysiological and performance measures and fuzzy clustering method

The human operator's ability to perform their tasks can fluctuate over time. Because the cognitive demands of the task can also vary it is possible that the capabilities of the operator are not sufficient to satisfy the job demands. This can lead to serious errors when the operator is overwhelmed by the task demands. Psychophysiological measures, such as heart rate and brain activity, can be used to monitor operator cognitive workload. In this paper, the most influential psychophysiological measures are extracted to characterize Operator Functional State (OFS) in automated tasks under a complex form of human-automation interaction. The fuzzy c-mean (FCM) algorithm is used and tested for its OFS classification performance. The results obtained have shown the feasibility and effectiveness of the FCM algorithm as well as the utility of the selected input features for OFS classification. Besides being able to cope with nonlinearity and fuzzy uncertainty in the psychophysiological data it can provide information about the relative importance of the input features as well as the confidence estimate of the classification results. The OFS pattern classification method developed can be incorporated into an adaptive aiding system in order to enhance the overall performance of a large class of safety-critical human-machine cooperative systems.

Coadaptive aiding and automation enhance operator performance

OBJECTIVE

In this work, we expand on the theory of adaptive aiding by measuring the effectiveness of coadaptive aiding, wherein we explicitly allow for both system and user to adapt to each other.

BACKGROUND

Adaptive aiding driven by psychophysiological monitoring has been demonstrated to be a highly effective means of controlling task allocation and system functioning. Psychophysiological monitoring is uniquely well suited for coadaptation, as malleable brain activity may be used as a continuous input to the adaptive system.

METHOD

To establish the efficacy of the coadaptive system, physiological activation of adaptation was directly compared with manual activation or no activation of the same automation and cuing systems. We used interface adaptations and automation that are plausible for real-world operations, presented in the context of a multi-remotely piloted aircraft control simulation. Each participant completed 3 days of testing during 1 week. Performance was assessed via proportion of targets successfully engaged.

RESULTS

In the first 2 days of testing, there were no significant differences in performance between the conditions. However, in the third session, physiological adaptation produced the highest performance.

CONCLUSION

By extending the data collection across multiple days, we offered enough time and repeated experience for user adaptation as well as online system adaptation, hence demonstrating coadaptive aiding.

APPLICATION

The results of this work may be employed to implement more effective adaptive workstations in a variety of work domains.

Operator functional state classification using least-square support vector machine based recursive feature elimination technique

This paper proposed two psychophysiological-data-driven classification frameworks for operator functional states (OFS) assessment in safety-critical human-machine systems with stable generalization ability. The recursive feature elimination (RFE) and least square support vector machine (LSSVM) are combined and used for binary and multiclass feature selection. Besides typical binary LSSVM classifiers for two-class OFS assessment, two multiclass classifiers based on multiclass LSSVM-RFE and decision directed acyclic graph (DDAG) scheme are developed, one used for recognizing the high mental workload and fatigued state while the other for differentiating overloaded and base-line states from the normal states. Feature selection results have revealed that different dimensions of OFS can be characterized by specific set of psychophysiological features. Performance comparison studies show that reasonable high and stable classification accuracy of both classification frameworks can be achieved if the RFE procedure is properly implemented and utilized.

An adaptive driver support system: user experiences and driving performance in a simulator

OBJECTIVE

The aim of this study was to test the implementation of an adaptive driver support system.

BACKGROUND

Providing support might not always be desirable from a safety perspective, as support may lead to problems related to a human operator being out of the loop. In contrast, adaptive support systems are designed to keep the operator in the loop as much as possible by providing support only when necessary.

METHOD

A total of 31 experienced drivers were exposed to three modes of lane-keeping support nonadaptive, adaptive, and no support. Support involved continuously updated lateral position feedback shown on a head-up display. When adaptive, support was triggered by performance-based indications of effort investment. Narrowing lane width and increasing density of oncoming traffic served to increase steering demand, and speed was fixed in all conditions to prevent any compensatory speed reactions.

RESULTS

Participants preferred the adaptive support mode mainly as a warning signal and tended to ignore nonadaptive feedback. Furthermore, driving behavior was improved by adaptive support in that participants drove more centrally, displayed less lateral variation and drove less outside the lane's delineation when support was in the adaptive mode compared with both the no-support mode and the nonadaptive support mode.

CONCLUSION

A human operator is likely to use machine-triggered adaptations as an indication that thresholds have been passed, regardless of the support that is initiated. Therefore supporting only the sensory processing stage of the human information processing system with adaptive automation may not feasible.

APPLICATION

These conclusions are relevant for designing adaptive driver support systems.

Diagnostic Monitoring Of Vigilance Decrement Using EEG Workload Indices

The resource model of vigilance (Warm, Parasuraman, & Matthews, 2008) suggests that EEG-based indices of workload might be used to monitor the operator’s fitness to sustain signal detection. 92 participants performed a 40 minute vigilance task believed to be sensitive to resource availability. Half performed in a cued condition, half without cues. Findings confirmed that cueing reduces workload and enhances vigilance. EEG was recorded throughout performance. Of the various EEG indices analyzed, lower frequency alpha and the Task Load Index (TLI) corresponded most closely to changes in signal detection rates. Other indices, the Engagement Index (EI) and frontal theta, did not show systematic decrement but discriminated cued and uncued conditions towards the end of the task. Implications of the findings for using EEG to drive adaptive automation are discussed.

Using scalp electrical biosignals to control an object by concentration and relaxation tasks: design and evaluation

In this paper we explore the use of electrical biosignals measured on scalp and corresponding to mental relaxation and concentration tasks in order to control an object in a video game. To evaluate the requirements of such a system in terms of sensors and signal processing we compare two designs. The first one uses only one scalp electroencephalographic (EEG) electrode and the power in the alpha frequency band. The second one uses sixteen scalp EEG electrodes and machine-learning methods. The role of muscular activity is also evaluated using five electrodes positioned on the face and the neck. Results show that the first design enabled 70% of the participants to successfully control the game, whereas 100% of the participants managed to do it with the second design based on machine learning. Subjective questionnaires confirm these results: users globally felt to have control in both designs, with an increased feeling of control in the second one. Offline analysis of face and neck muscle activity shows that this activity could also be used to distinguish between relaxation and concentration tasks. Results suggest that the combination of muscular and brain activity could improve performance of this kind of system. They also suggest that muscular activity has probably been recorded by EEG electrodes.