Biocybernetic system evaluates indices of operator engagement in automated task

QUASAR's QStates cognitive gauge performance in the cognitive state assessment competition 2011

The Cognitive State Assessment Competition 2011 was organized by the U.S. Air Force Research Laboratory (AFRL) to compare the performance of real-time cognitive state classification software. This paper presents results for QUASAR's data classification module, QStates, which is a software package for real-time (and off-line) analysis of physiologic data collected during cognitive-specific tasks. The classifier's methodology can be generalized to any particular cognitive state; QStates identifies the most salient features extracted from EEG signals recorded during different cognitive states or loads.

Preliminary results of mental workload and task engagement assessment using electroencephalogram in a space suit

In this paper, we present preliminary results of subject's mental workload and task engagement assessment in an experimental space suit. We have quantified the mental workload and task engagement based on changes in electroencephalogram (EEG). EEG signals were collected from subjects scalp using a commercial wireless EEG device in two experimental conditions - when subjects did not wear space suit (control condition) and when subjects wore space suit. Brain state changes were estimated and compared with the direct responses for different tasks and different conditions. We found that the spacesuit experiment introduced a greater mental workload where subject's stress levels were higher than control experiment.

Neurocognitive and Psychophysiological Interfaces for Adaptive Virtual Environments

The use of neuropsychological and psychophysiological measures in studies of patients immersed in high-fidelity virtual environments offers the potential to develop current psychophysiological computing approaches into affective computing scenarios that can be used for assessment, diagnosis and treatment planning. Such scenarios offer the potential for simulated environments to proffer cogent and calculated response approaches to real-time changes in user emotion, neurocognition, and motivation. The value in using virtual environments to produce simulations targeting these areas has been acknowledged by an encouraging body of research. Herein the authors describe (1) literature on virtual environments for neurocognitive and psychophysiological profiles of users’ individual strengths and weaknesses; and (2) real-time adaptation of virtual environments that could be used for virtual reality exposure therapy and cognitive rehabilitation. Specifically, the authors discuss their approach to an adaptive environment that uses the principles of flow, presence, neuropsychology, psychophysiology to develop a novel application for rehabilitative applications.

Human Supervisory Control Challenges in Network-Centric Operations

Network-centric operations (NCOs), envisioned for future command and control systems in military and civilian settings, must be supported by sophisticated automated systems, so human-computer interactions are an important aspect of overall system performance. This chapter identifies 10 human supervisory control challenges that could significantly impact operator performance in NCOs: information overload, attention allocation, decision biases, supervisory monitoring of operators, distributed decision making through team coordination, trust and reliability, the role of automation, adaptive automation, multimodal technologies, and accountability. NCOs will bring increases in the number of information sources, volume of information, and operational tempo with significant uncertainty, all of which will place higher cognitive demands on operators. Thus, it is critical that NCO research focuses not only on technological innovations but also on the strengths and limitations of human-automation interaction in a complex system.

Combining electrodermal responses and cardiovascular measures for probing adaptive automation during simulated flight

Adaptive automation increases the operator's workload in case of hypovigilance and takes over more responsibility if workload becomes too high. Two consecutive studies were conducted to construct a biocybernetic adaptive system for a professional flight simulator, based on autonomic measures. Workload was varied through different stages of turbulences. In a first study with 18 participants, electrodermal responses of experimental subjects oscillated very close to the individual set point, demonstrating that workload level was adjusted as a result of adaptive control, which was not the case in yoked control subjects without adaptive automation. Combining electrodermal responses with heart rate variability in a second study with 48 participants further enhanced the adaptive power which was seen in even smaller set point deviations for the experimental compared to the yoked control group. We conclude that the level of arousal can be adjusted to avoid hypovigilance by combining autonomic measures in a closed loop.

Sensitivity of candidate markers of psychophysiological strain to cyclical changes in manual control load during simulated process control

Complex systems are vulnerable to unpredictable breakdowns in operator performance. Although primary task goals are typically protected by compensatory effort, such protection may break down under fatigue and high strain. Detection of strain states would enable prediction of increased operational risk through adaptive automation, triggering a switch of control from human to computer. A simulated process control task was used to identify markers of strain under a cyclic loading procedure, which forced performance breakdown through stepwise changes in control load. Four trained participants provided data on control performance and a range of candidate psychophysiological markers of strain (two EEG power ratios and HRV). Within-individual analyses showed the strongest sensitivity for 'task load index' (TLI), an EEG measure based on executive control activity in frontal brain areas, though all measures were sensitive for some participants. The implications of such findings for the development of a closed loop system for adaptive automation are discussed.

Cardiovascular state changes during performance of a simulated ambulance dispatchers' task: potential use for adaptive support

Adaptive support has the potential to keep the operator optimally motivated, involved, and able to perform a task. In order to use such support, the operator's state has to be determined from physiological parameters and task performance measures. In an environment where the task of an ambulance dispatcher was simulated, two studies have been carried out to evaluate the feasibility of using cardiovascular measures for adaptive support. During performance of this 2-3h lasting planning task, a pattern of results is found that can be characterized by an initial increase of blood pressure and heart rate and a decrease of heart rate variability (defense reaction pattern) followed by an ongoing increase of blood pressure counteracted by a decrease in heart rate. This pattern can be explained by an augmented short-term blood pressure control (baroreflex), which is reflected in an increase of baroreflex sensitivity. Additionally, in this latter phase heart rate variability (HRV) increases as a function of time, while blood pressure variability decreases. In the two studies performed, the baroreflex pattern was consistent for all the relevant variables. In both studies there were periods with high and low workload. Effects of task load are mainly reflected in the variability measures, while in the second study, additionally, blood pressure level was higher during periods with high task demands. The conclusion of the studies is that consistent cardiovascular response patterns can be recognized during this semi-realistic planning task, where variability measures are most sensitive to task demand changes, while blood pressure and baroreflex sensitivity are most informative with respect to cardiovascular state changes. These findings can be seen as a great potential benefit for future use in adaptive support applications.

Augmented Cognition: An Overview

Augmented cognition is a form of human-systems interaction in which a tight coupling between user and computer is achieved via physiological and neurophysiological sensing of a user's cognitive state. This interactive paradigm seeks to revolutionize the manner in which humans engage with computers by leveraging this knowledge of cognitive state to precisely adapt user-system interaction in real time. This review provides an overview of contemporary works in the field of augmented cognition and details regarding the three main components of an augmented cognition system: cognitive state sensors, adaptation strategies, and control systems. The review provides a perspective on the field as well as insights into the many challenges that lie ahead for those who endeavor to realize the full potential of augmented cognition.

Performance enhancement in an uninhabited air vehicle task using psychophysiologically determined adaptive aiding

OBJECTIVE

We show that psychophysiologically driven real-time adaptive aiding significantly enhances performance in a complex aviation task. A further goal was to assess the importance of individual operator capabilities when providing adaptive aiding.

BACKGROUND

Psychophysiological measures are useful for monitoring cognitive workload in laboratory and real-world settings. They can be recorded without intruding into task performance and can be analyzed in real time, making them candidates for providing operator functional state estimates. These estimates could be used to determine if and when system intervention should be provided to assist the operator to improve system performance.

METHODS

Adaptive automation was implemented while operators performed an uninhabited aerial vehicle task. Psychophysiological data were collected and an artificial neural network was used to detect periods of high and low mental workload in real time. The high-difficulty task levels used to initiate the adaptive automation were determined separately for each operator, and a group-derived mean difficulty level was also used.

RESULTS

Psychophysiologically determined aiding significantly improved performance when compared with the no-aiding conditions. Improvement was greater when adaptive aiding was provided based on individualized criteria rather than on group-derived criteria. The improvements were significantly greater than when the aiding was randomly provided.

CONCLUSION

These results show that psychophysiologically determined operator functional state assessment in real time led to performance improvement when included in closed loop adaptive automation with a complex task.

APPLICATION

Potential future applications of this research include enhanced workstations using adaptive aiding that would be driven by operator functional state.

Operator Engagement Detection for Robot Behavior Adaptation

It has been shown that in human-robot interaction, the effectiveness of a robot varies inversely with the operator engagement in the task. Given the importance of maintaining optimal task engagement when working with a robot, it would be immensely useful to have a robotic system that can detect the level of operator engagement and modify its behavior if required. This paper presents a framework for human-robot interaction that allows inference of operator's engagement level through the analysis of his/her physiological signals, and adaptation of robot behavior as a function of the operator's engagement level. Peripheral physiological signals were measured through wearable biofeedback sensors and a control architecture inspired by Riley's original information-flow model was developed to implement such human-robot interaction. The results from affect-elicitation tasks for human participants showed that it was possible to detect engagement through physiological sensing in real-time. An open-loop teleoperation-based robotic experiment was also conducted where the recorded physiological signals were transmitted to the robot in real-time speed to demonstrate that the presented control architecture allowed the robot to adapt its behavior based on operator engagement level.

Comparison of a brain-based adaptive system and a manual adaptable system for invoking automation

OBJECTIVE

Two experiments are presented examining adaptive and adaptable methods for invoking automation.

BACKGROUND

Empirical investigations of adaptive automation have focused on methods used to invoke automation or on automation-related performance implications. However, no research has addressed whether performance benefits associated with brain-based systems exceed those in which users have control over task allocations.

METHOD

Participants performed monitoring and resource management tasks as well as a tracking task that shifted between automatic and manual modes. In the first experiment, participants worked with an adaptive system that used their electroencephalographic signals to switch the tracking task between automatic and manual modes. Participants were also divided between high- and low-reliability conditions for the system-monitoring task as well as high- and low-complacency potential. For the second experiment, participants operated an adaptable system that gave them manual control over task allocations.

RESULTS

Results indicated increased situation awareness (SA) of gauge instrument settings for individuals high in complacency potential using the adaptive system. In addition, participants who had control over automation performed more poorly on the resource management task and reported higher levels of workload. A comparison between systems also revealed enhanced SA of gauge instrument settings and decreased workload in the adaptive condition.

CONCLUSION

The present results suggest that brain-based adaptive automation systems may enhance perceptual level SA while reducing mental workload relative to systems requiring user-initiated control.

APPLICATION

Potential applications include automated systems for which operator monitoring performance and high-workload conditions are of concern.

Prediction of subjective states from psychophysiology: A multivariate approach

Biocybernetic systems utilise real-time changes in psychophysiology in order to adapt aspects of computer control and functionality, e.g. adaptive automation. This approach to system design is based upon an assumption that psychophysiological variations represent implicit fluctuations in the subjective state of the operator, e.g. mood, motivation, cognitions. A study was performed to investigate the convergent validity between psychophysiological measurement and changes in the subjective status of the individual. Thirty-five participants performed a demanding version of the Multi-Attribute Task Battery (MATB) over four consecutive 20-min blocks. A range of psychophysiological data were collected (EEG, ECG, skin conductance level (SCL), EOG, respiratory rate) and correlated with changes in subjective state as measured by the Dundee Stress State Questionnaire (DSSQ). MATB performance was stable across time-on-task; psychophysiological activity exhibited expected changes due to sustained performance. The DSSQ was analysed in terms of three subjective meta-factors: Task Engagement, Distress and Worry. Multiple regression analyses revealed that psychophysiology predicted a substantial proportion of the variance for both Task Engagement and Distress but not for the Worry meta-factor. The consequences for the development of biocybernetic systems are discussed.

The influence of task demand and learning on the psychophysiological response

The level of expertise of an operator may significantly influence his/her psychophysiological response to high task demand. A naive individual may invest considerable mental effort during performance of a difficult task and psychophysiological reactivity will be high compared to the psychophysiological response of a highly skilled operator. A study on multitasking performance was conducted to investigate the interaction between learning and task demand on psychophysiological reactivity. Thirty naive participants performed high and low demand versions of the Multi-attribute Task Battery (MATB) over a learning period of 64 min. High and low task demand setting were preset via a pilot study. Psychophysiological variables were collected from four channels of EEG (Cz, P3, P4, Pz), ECG, EOG and respiration rate to measure the impact of task demand and learning. Several variables were sensitive to the task demand manipulation but not time-on-task, e.g., heart rate, Theta activity at parietal sites. The sensitivity of certain variables to high demand was compromised by skill acquisition, e.g., respiration rate, suppression of alpha activity. A sustained learning effect was observed during the high demand condition only; multiple regression analyses revealed that specific psychophysiological variables predicted learning at different stages on the learning curve. The implications for the sensitivity of psychophysiological variables are discussed.

An evaluation of an adaptive automation system using a cognitive vigilance task

The performance of an adaptive automation system was evaluated using a cognitive vigilance task. Participants responded to the presence of a green "K" in an array of two, five, or nine distractor stimuli during a 40-min vigil. The array with the target stimulus was presented once each minute. Participants EEG was recorded and an engagement index (EI = 20 x beta/(alpha + theta)) was derived. In the negative feedback condition, increases in the EI caused the number of stimuli in the array to decrease while decreases in the EI caused the number of stimuli to increase. For the positive feedback condition, increases in the index caused an increase in the array size (AS) while decreases caused a decrease in the array size. Each experimental participant had a yoked control partner who received the same pattern of changes in array irrespective of their engagement index. A vigilance decrement was seen only for the positive feedback, experimental group.

Real-time assessment of mental workload using psychophysiological measures and artificial neural networks

The functional state of the human operator is critical to optimal system performance. Degraded states of operator functioning can lead to errors and overall suboptimal system performance. Accurate assessment of operator functional state is crucial to the successful implementation of an adaptive aiding system. One method of determining operators' functional state is by monitoring their physiology. In the present study, artificial neural networks using physiological signals were used to continuously monitor, in real time, the functional state of 7 participants while they performed the Multi-Attribute Task Battery with two levels of task difficulty. Six channels of brain electrical activity and eye, heart and respiration measures were evaluated on line. The accuracy of the classifier was determined to test its utility as an on-line measure of operator state. The mean classification accuracies were 85%, 82%, and 86% for the baseline, low task difficulty, and high task difficulty conditions, respectively. The high levels of accuracy suggest that these procedures can be used to provide accurate estimates of operator functional state that can be used to provide adaptive aiding. The relative contribution of each of the 43 psychophysiological features was also determined. Actual or potential applications of this research include test and evaluation and adaptive aiding implementation.

Effects of a psychophysiological system for adaptive automation on performance, workload, and the event-related potential P300 component

The present study examined the effects of an electroencephalographic- (EEG-) based system for adaptive automation on tracking performance and workload. In addition, event-related potentials (ERPs) to a secondary task were derived to determine whether they would provide an additional degree of workload specificity. Participants were run in an adaptive automation condition, in which the system switched between manual and automatic task modes based on the value of each individual's own EEG engagement index; a yoked control condition; or another control group, in which task mode switches followed a random pattern. Adaptive automation improved performance and resulted in lower levels of workload. Further, the P300 component of the ERP paralleled the sensitivity to task demands of the performance and subjective measures across conditions. These results indicate that it is possible to improve performance with a psychophysiological adaptive automation system and that ERPs may provide an alternative means for distinguishing among levels of cognitive task demand in such systems. Actual or potential applications of this research include improved methods for assessing operator workload and performance.

Operator functional state classification using multiple psychophysiological features in an air traffic control task

We studied 2 classifiers to determine their ability to discriminate among 4 levels of mental workload during a simulated air traffic control task using psychophysiological measures. Data from 7 air traffic controllers were used to train and test artificial neural network and stepwise discriminant classifiers. Very high levels of classification accuracy were achieved by both classifiers. When the 2 task difficulty manipulations were tested separately, the percentage correct classifications were between 84% and 88%. Feature reduction using saliency analysis for the artificial neural networks resulted in a mean of 90% correct classification accuracy. Considering the data as a 2-class problem, acceptable load versus overload, resulted in almost perfect classification accuracies, with mean percentage correct of 98%. In applied situations, the most important distinction among operator functional states would be to detect mental overload situations. These results suggest that psychophysiological data are capable of such discriminations with high levels of accuracy. Potential applications of this research include test and evaluation of new and modified systems and adaptive aiding.

Effects of a biocybernetic system on vigilance performance

The present study was designed to determine whether a biocybernetic, adaptive system could enhance vigilance performance. Participants were asked to monitor the repetitive presentation of white bars on a computer screen for occasional increases in length. An index of task engagement was derived from participants' electroencephalographic (EEG) activity and was used to change the presentation rate of events among 3 values (6, 20, and 60 events/min). Under a negative feedback contingency, event rates increased if the engagement index decreased and, conversely, decreased if the index increased. Under positive feedback, the opposite contingency existed. Each experimental participant had a yoked control partner who received the same pattern of changes in event rates irrespective of his or her EEG activity. The results showed that better vigilance performance was obtained under negative feedback and that the performance of the yoked participants was similar to that of their experimental partners. These findings suggest that it may be possible to improve monitoring performance on critical activities such as air traffic control and radar and sonar operation through a pattern of event rate changes that do not rely on an operator's overt behavior.

Self-treatment of hypoglycemia while driving

OBJECTIVE

While it is clear that progressive diabetic hypoglycemia leads to neuroglycopenia, which impairs driving, it is not clear what contributes to patients' detection and subsequent self-correction of hypoglycemia/driving impairments. Drivers with Type 1 Diabetes Mellitus (T1DM) who did and did not engage in self-treatment during experimental hypoglycemia driving are compared physiologically and psychologically.

METHOD

38 drivers with T1DM drove a sophisticated driving simulator during euglycemia and progressive hypoglycemia. Subjects were continually monitored for driving performance, EEG activity and whether they self-treated with a glucose drink. Every 5 min measures were taken of blood glucose (BG) and epinephrine levels, perceived neurogenic and neuroglycopenic symptoms and driving ability. For the four weeks prior to this hospital study, subjects participated in a field study. Using a hand-held computer just prior to routine self-measurements of BG, subjects rated neurogenic and neuroglycopenic symptoms and made judgements about BG level and ability to drive as they did in the hospital.

RESULTS

Drivers who did and did not self-treat did not differ in terms of their pre-hospital exposure to hypoglycemia, their depth and rate of BG fall during experimental testing, or their epinephrine response to hypoglycemia. Subjects who self-treated detected more neurogenic and neuroglycopenic symptoms than those who did not self-treat. They also experienced less EEG defined neuroglycopenia during the progressive hypoglycemic drive as compared to those who did not self-treat. Perceived need to self-treat and EEG parameters correctly classified 88% of those who did treat from those who did not self-treat. Further, subjects who self-treated were more aware of hypoglycemia and when not to drive while hypoglycemic in the field study.

CONCLUSION

There is a narrow window between a patient's detection of hypoglycemic symptoms and the need to self-treat, and neuroglycopenia, which impairs self-treatment. Consequently, drivers with T1DM should be vigilant for signs of hypoglycemia and driving impairment (e.g. trembling, uncoordination, visual difficulties) and encouraged to treat themselves immediately when they suspect hypoglycemia while driving.

A closed-loop system for examining psychophysiological measures for adaptive task allocation

A closed-loop system was evaluated for its efficacy in using psychophysiological indexes to moderate workload. Participants were asked to perform either 1 or 3 tasks from the Multiattribute Task Battery and complete the NASA Task Load Index after each trial. An electroencephalogram (EEG) was sampled continuously while they performed the tasks, and an EEG index (beta/alpha plus theta) was derived. The system made allocation decisions as a function of the level of operator engagement based on the value of the EEG index. The results of the study demonstrated that it was possible to moderate an operator's level of engagement through a closed-loop system driven by the operator's own EEG. In addition, the system had a significant impact on behavioral, subjective, and psychophysiological correlates of workload as task load increased. The theoretical and practical implications of these results for adaptive automation are discussed.

Evaluation of an adaptive automation system using three EEG indices with a visual tracking task

A system was evaluated for use in adaptive automation using two experiments with electroencephalogram (EEG) indices based on the beta, alpha, and theta bandwidths. Subjects performed a compensatory tracking task while their EEG was recorded and converted to one of three engagement indices: beta/(alpha + theta), beta/alpha, or 1/alpha. In experiment one, the tracking task was switched between manual and automatic modes depending on whether the subject's engagement index was increasing or decreasing under a positive or negative feedback condition. Subjects were run for three consecutive 16-min trials. In experiment two, the task was switched depending on whether the absolute level of the engagement index for the subject was above or below baseline levels. It was hypothesized that negative feedback would produce more switches between manual and automatic modes, and that the beta/(alpha + theta) index would be most effective. The results confirmed these hypotheses. Tracking performance was better under negative feedback in both experiments; also, the use of absolute levels of engagement in experiment two resulted in better performance. There were no systematic changes in these effects over three 16-min trials. The implications for the use of such systems for adaptive automation are discussed.

Comparison of two methods for use in adaptive automation

Two psychophysiological adaptive automation methods were compared to assess their relative influence on teaching performance of 16 adults.

Psychophysiology and adaptive automation

Adaptive automation is an approach to automation design where tasks are dynamically allocated between the human operator and computer systems. Psychophysiology has two complementary roles in research on adaptive automation: first, to provide information about the effects of different forms of automation thus promoting the development of effective adaptive logic; and second, psychophysiology may yield information about the operator that can be integrated with performance measurement and operator modelling to aid in the regulation of automation. This review discusses the basic tenets of adaptive automation and the role of psychophysiological measures in the study of adaptive automation. Empirical results from studies of flight simulation are presented. Psychophysiological measures may prove especially useful in the prevention of performance deterioration in underload conditions that may accompany automation. Individual differences and the potential for learned responses require research to understand their influence on adaptive algorithms. Adaptive automation represents a unique domain for the application of psychophysiology in the work environment.