Biocybernetic system evaluates indices of operator engagement in automated task

Enhancing Sustained Attention

Arguably, automation is fast transforming many enterprise business processes, transforming operational jobs into monitoring tasks. Consequently, the ability to sustain attention during extended periods of monitoring is becoming a critical skill. This manuscript presents a Brain-Computer Interface (BCI) prototype which seeks to combat decrements in sustained attention during monitoring tasks within an enterprise system. A brain-computer interface is a system which uses physiological signals output by the user as an input. The goal is to better understand human responses while performing tasks involving decision and monitoring cycles, finding ways to improve performance and decrease on-task error. Decision readiness and the ability to synthesize complex and abundant information in a brief period during critical events has never been more important. Closed-loop control and motivational control theory were synthesized to provide the basis from which a framework for a prototype was developed to demonstrate the feasibility and value of a BCI in critical enterprise activities. In this pilot study, the BCI was implemented and evaluated through laboratory experimentation using an ecologically valid task. The results show that the technological artifact allowed users to regulate sustained attention positively while performing the task. Levels of sustained attention were shown to be higher in the conditions assisted by the BCI. Furthermore, this increased cognitive response seems to be related to increased on-task action and a small reduction in on-task errors. The research concludes with a discussion of the future research directions and their application in the enterprise.

From Textbook to Teacher: an Adaptive Intelligent Tutoring System Based on BCI

In this work, we propose FT³, an adaptive intelligent tutoring system based on Brain Computer Interface(BCI). It can automatically generate different difficulty levels of lecturing video with teachers from textbook adapting to student engagement measured by BCI. Most current studies employ animated images to create pedagogical agents in such adaptive learning environments. However, evidence suggests that human teacher video brings a better learning experience than animated images. We design a virtual teacher generation engine consisting of text-to-speech (TTS) and lip synthesis method, being able to generate high-quality adaptive lecturing clips of talking teachers with accurate lip sync merely based on a textbook and teacher's photo. We propose a BCI to measure engagement, serving as an indicator for adaptively generating appropriate lecturing videos. We conduct a preliminary study to build and evaluate FT³. Results verify that FT3 can generate synced lecturing videos, and provide proper levels of learning content with an accuracy of 73.33%.

Mental Workload Estimation Based on Physiological Features for Pilot-UAV Teaming Applications

Manned-Unmanned Teaming (MUM-T) can be defined as the teaming of aerial robots (artificial agents) along with a human pilot (natural agent), in which the human agent is not an authoritative controller but rather a cooperative team player. To our knowledge, no study has yet evaluated the impact of MUM-T scenarios on operators' mental workload (MW) using a neuroergonomic approach (i.e., using physiological measures), nor provided a MW estimation through classification applied on those measures. Moreover, the impact of the non-stationarity of the physiological signal is seldom taken into account in classification pipelines, particularly regarding the validation design. Therefore this study was designed with two goals: (i) to characterize and estimate MW in a MUM-T setting based on physiological signals; (ii) to assess the impact of the validation procedure on classification accuracy. In this context, a search and rescue (S&R) scenario was developed in which 14 participants played the role of a pilot cooperating with three UAVs (Unmanned Aerial Vehicles). Missions were designed to induce high and low MW levels, which were evaluated using self-reported, behavioral and physiological measures (i.e., cerebral, cardiac, and oculomotor features). Supervised classification pipelines based on various combinations of these physiological features were benchmarked, and two validation procedures were compared (i.e., a traditional one that does not take time into account vs. an ecological one that does). The main results are: (i) a significant impact of MW on all measures, (ii) a higher intra-subject classification accuracy (75%) reached using ECG features alone or in combination with EEG and ET ones with the Adaboost, Linear Discriminant Analysis or the Support Vector Machine classifiers. However this was only true with the traditional validation. There was a significant drop in classification accuracy using the ecological one. Interestingly, inter-subject classification with ecological validation (59.8%) surpassed both intra-subject with ecological and inter-subject with traditional validation. These results highlight the need for further developments to perform MW monitoring in such operational contexts.

Impact of screen size on cognitive training task performance: An HMD study

To better understand the impact of different screen sizes in cognitive training, study subjects performed an adaptive training task at three separate visual angles using a head-mounted display (HMD). Cognitive load was assessed using EEG and compared with task performance (accuracy and response time) for each condition. While previous studies found performance benefits corresponding to increased screen size in memory and learning tasks, our results suggest such benefits may only apply up to a visual angle of approximately 20°, after which increases in size become inversely correlated with task performance.

Real-time prediction of short-timescale fluctuations in cognitive workload

Human operators often experience large fluctuations in cognitive workload over seconds timescales that can lead to sub-optimal performance, ranging from overload to neglect. Adaptive automation could potentially address this issue, but to do so it needs to be aware of real-time changes in operators' spare cognitive capacity, so it can provide help in times of peak demand and take advantage of troughs to elicit operator engagement. However, it is unclear whether rapid changes in task demands are reflected in similarly rapid fluctuations in spare capacity, and if so what aspects of responses to those demands are predictive of the current level of spare capacity. We used the ISO standard detection response task (DRT) to measure cognitive workload approximately every 4 s in a demanding task requiring monitoring and refueling of a fleet of simulated unmanned aerial vehicles (UAVs). We showed that the DRT provided a valid measure that can detect differences in workload due to changes in the number of UAVs. We used cross-validation to assess whether measures related to task performance immediately preceding the DRT could predict detection performance as a proxy for cognitive workload. Although the simple occurrence of task events had weak predictive ability, composite measures that tapped operators' situational awareness with respect to fuel levels were much more effective. We conclude that cognitive workload does vary rapidly as a function of recent task events, and that real-time predictive models of operators' cognitive workload provide a potential avenue for automation to adapt without an ongoing need for intrusive workload measurements.

Can EEG Be Adopted as a Neuroscience Reference for Assessing Software Programmers' Cognitive Load?

An emergent research area in software engineering and software reliability is the use of wearable biosensors to monitor the cognitive state of software developers during software development tasks. The goal is to gather physiologic manifestations that can be linked to error-prone scenarios related to programmers’ cognitive states. In this paper we investigate whether electroencephalography (EEG) can be applied to accurately identify programmers’ cognitive load associated with the comprehension of code with different complexity levels. Therefore, a controlled experiment involving 26 programmers was carried. We found that features related to Theta, Alpha, and Beta brain waves have the highest discriminative power, allowing the identification of code lines and demanding higher mental effort. The EEG results reveal evidence of mental effort saturation as code complexity increases. Conversely, the classic software complexity metrics do not accurately represent the mental effort involved in code comprehension. Finally, EEG is proposed as a reference, in particular, the combination of EEG with eye tracking information allows for an accurate identification of code lines that correspond to peaks of cognitive load, providing a reference to help in the future evaluation of the space and time accuracy of programmers’ cognitive state monitored using wearable devices compatible with software development activities.

The Influence of Video Format on Engagement and Performance in Online Learning

Millions of students follow online classes which are delivered in video format. Several studies examine the impact of these video formats on engagement and learning using explicit measures and outline the need to also investigate the implicit cognitive and emotional states of online learners. Our study compared two video formats in terms of engagement (over time) and learning in a between-subject experiment. Engagement was operationalized using explicit and implicit neurophysiological measures. Twenty-six (26) subjects participated in the study and were randomly assigned to one of two conditions based on the video shown: infographic video or lecture capture. The infographic video showed animated graphics, images, and text. The lecture capture showed a professor, providing a lecture, filmed in a classroom setting. Results suggest that lecture capture triggers greater emotional engagement over a shorter period, whereas the infographic video maintains higher emotional and cognitive engagement over longer periods of time. Regarding student learning, the infographic video contributes to significantly improved performance in matters of difficult questions. Additionally, our results suggest a significant relationship between engagement and student performance. In general, the higher the engagement, the better the student performance, although, in the case of cognitive engagement, the link is quadratic (inverted U shaped).

Sensor-based indicators of performance changes between sessions during robotic surgery training

Training of surgeons is essential for safe and effective use of robotic surgery, yet current assessment tools for learning progression are limited. The objective of this study was to measure changes in trainees' cognitive and behavioral states as they progressed in a robotic surgeon training curriculum at a medical institution. Seven surgical trainees in urology who had no formal robotic training experience participated in the simulation curriculum. They performed 12 robotic skills exercises with varying levels of difficulty repetitively in separate sessions. EEG (electroencephalogram) activity and eye movements were measured throughout to calculate three metrics: engagement index (indicator of task engagement), pupil diameter (indicator of mental workload) and gaze entropy (indicator of randomness in gaze pattern). Performance scores (completion of task goals) and mental workload ratings (NASA-Task Load Index) were collected after each exercise. Changes in performance scores between training sessions were calculated. Analysis of variance, repeated measures correlation, and machine learning classification were used to diagnose how cognitive and behavioral states associate with performance increases or decreases between sessions. The changes in performance were correlated with changes in engagement index (rrm=-.25,p<.001) and gaze entropy (rrm=-.37,p<.001). Changes in cognitive and behavioral states were able to predict training outcomes with 72.5% accuracy. Findings suggest that cognitive and behavioral metrics correlate with changes in performance between sessions. These measures can complement current feedback tools used by medical educators and learners for skills assessment in robotic surgery training.

How Can Physiological Computing Benefit Human-Robot Interaction?

As systems grow more automatized, the human operator is all too often overlooked. Although human-robot interaction (HRI) can be quite demanding in terms of cognitive resources, the mental states (MS) of the operators are not yet taken into account by existing systems. As humans are no providential agents, this lack can lead to hazardous situations. The growing number of neurophysiology and machine learning tools now allows for efficient operators’ MS monitoring. Sending feedback on MS in a closed-loop solution is therefore at hand. Involving a consistent automated planning technique to handle such a process could be a significant asset. This perspective article was meant to provide the reader with a synthesis of the significant literature with a view to implementing systems that adapt to the operator’s MS to improve human-robot operations’ safety and performance. First of all, the need for this approach is detailed regarding remote operation, an example of HRI. Then, several MS identified as crucial for this type of HRI are defined, along with relevant electrophysiological markers. A focus is made on prime degraded MS linked to time-on-task and task demands, as well as collateral MS linked to system outputs (i.e., feedback and alarms). Lastly, the principle of symbiotic HRI is detailed and one solution is proposed to include the operator state vector into the system using a mixed-initiative decisional framework to drive such an interaction.

Cardiac sympathetic activity during recovery as an indicator of sympathetic activity during task performance

The goals of this research were to analyze cardiac sympathetic recovery patterns and evaluate whether sympathetic cardiac responses to a task challenge can be predicted using residual cardiac activity measured directly after the task (that is, during the recovery period). In two studies (total N = 181), we measured cardiac sympathetic activity, quantified as pre-ejection period and RB interval, during both task performance and the 2-min recovery period following the task. Additional analyses examined effects on the RZ interval. We found that sympathetic recovery from a task was rather quick: Cardiovascular recovery occurred within the first 30 s of the recovery period. Nevertheless, residual cardiac activity during the recovery period had predictive power for task-related cardiac activity. This suggests that sympathetic cardiac activity during recovery may serve as a useful indicator of task-related cardiac sympathetic activity. We discuss the implications of these findings for practical applications and the design of future studies.

Tracing Pilots' Situation Assessment by Neuroadaptive Cognitive Modeling

This study presents the integration of a passive brain-computer interface (pBCI) and cognitive modeling as a method to trace pilots' perception and processing of auditory alerts and messages during operations. Missing alerts on the flight deck can result in out-of-the-loop problems that can lead to accidents. By tracing pilots' perception and responses to alerts, cognitive assistance can be provided based on individual needs to ensure they maintain adequate situation awareness. Data from 24 participating aircrew in a simulated flight study that included multiple alerts and air traffic control messages in single pilot setup are presented. A classifier was trained to identify pilots' neurophysiological reactions to alerts and messages from participants' electroencephalogram (EEG). A neuroadaptive ACT-R model using EEG data was compared to a conventional normative model regarding accuracy in representing individual pilots. Results show that passive BCI can distinguish between alerts that are processed by the pilot as task-relevant or irrelevant in the cockpit based on the recorded EEG. The neuroadaptive model's integration of this data resulted in significantly higher performance of 87% overall accuracy in representing individual pilots' responses to alerts and messages compared to 72% accuracy of a normative model that did not consider EEG data. We conclude that neuroadaptive technology allows for implicit measurement and tracing of pilots' perception and processing of alerts on the flight deck. Careful handling of uncertainties inherent to passive BCI and cognitive modeling shows how the representation of pilot cognitive states can be improved iteratively for providing assistance.

A Neuroergonomics Approach to Mental Workload, Engagement and Human Performance

The assessment and prediction of cognitive performance is a key issue for any discipline concerned with human operators in the context of safety-critical behavior. Most of the research has focused on the measurement of mental workload but this construct remains difficult to operationalize despite decades of research on the topic. Recent advances in Neuroergonomics have expanded our understanding of neurocognitive processes across different operational domains. We provide a framework to disentangle those neural mechanisms that underpin the relationship between task demand, arousal, mental workload and human performance. This approach advocates targeting those specific mental states that precede a reduction of performance efficacy. A number of undesirable neurocognitive states (mind wandering, effort withdrawal, perseveration, inattentional phenomena) are identified and mapped within a two-dimensional conceptual space encompassing task engagement and arousal. We argue that monitoring the prefrontal cortex and its deactivation can index a generic shift from a nominal operational state to an impaired one where performance is likely to degrade. Neurophysiological, physiological and behavioral markers that specifically account for these states are identified. We then propose a typology of neuroadaptive countermeasures to mitigate these undesirable mental states.

Studying the Effect of Display Type and Viewing Perspective on User Experience in Virtual Reality Exergames

Background: Physical inactivity has been identified as the fourth leading cause of death globally. It is now well established that a sedentary lifestyle is a unique risk factor for several diseases such as type 2 diabetes and cardiovascular disease, which account for about 30% of global mortality. Diabetes is a major preventable cause of costly and debilitating renal failure, heart disease, lower limb amputation, and avoidable blindness. In recent years, the idea of using interactive computing systems that leverage gamification to promote physical activity has been widely researched. Prior studies have shown that exergames, that is those that encourage physical activity, can increase enjoyment and intrinsic motivation compared with conventional exercises; as such, they can be effective in promoting physical and mental health. There has been some research on immersive virtual reality (VR) exergames; however, to the best of our knowledge, it is limited and preliminary. This work aims at filling the gap and investigates the effect of display type (DT) and viewing perspective (VP) on players' exertion, engagement, and overall game experience in immersive VR exergames. Objective: This article aims at examining whether DT and VP can affect gameplay performance, players' exertion, game experience, cybersickness, and electroencephalography (EEG) engagement index when playing a gesture-based (i.e., body motion) exergame. Materials and Methods: Study 1 employed a one-way between-subjects design with 24 participants equally distributed in two groups (immersive VR and 50-inch TV) to perform 12 pre-defined gestures. The main outcome measures were National Aeronautics and Space Administration-Task Load Index (NASA-TLX) workload for each group as well as 7 Likert scale and EEG engagement index for each gesture. Study 2 included 16 participants in playing a game with the gestures selected from study 1. All participants played 4 versions based on combinations of DT (immersive VR and 50-inch TV) and VP (first-person and third-person) to assess exertion (%HR_max, calories consumption, and Borg RPE 6-20), game experience, cybersickness, and EEG engagement index. Results: Study 1 results showed that DT had no effect on the ratings of the gestures, NASA-TLX workload, and EEG engagement index. Study 2 results showed that immersive VR not only resulted in a significantly higher exertion (%HR_max, calories consumption, and Borg RPE) but also helped achieve better positive game experience in challenge, flow, sensory and imaginative immersion, as well as lower negative affect. We also found that nausea and oculomotor were significantly higher in immersive VR. Conclusion: This pilot study demonstrates that youth who played gesture-based exergame in immersive VR had a higher level of exertion (%HR_max, calories consumption, and Borg RPE), although the number of performed gestures were not significantly different. They also felt that immersive VR was much more challenging, immersive (flow, sensory and imaginative immersion), and had a lower negative affect than a 50-inch TV; however, immersive VR was more likely to make youth have higher cybersickness.

Electroencephalographic Workload Indicators During Teleoperation of an Unmanned Aerial Vehicle Shepherding a Swarm of Unmanned Ground Vehicles in Contested Environments

Background: Although many electroencephalographic (EEG) indicators have been proposed in the literature, it is unclear which of the power bands and various indices are best as indicators of mental workload. Spectral powers (Theta, Alpha, and Beta) and ratios (Beta/(Alpha + Theta), Theta/Alpha, Theta/Beta) were identified in the literature as prominent indicators of cognitive workload.

Objective: The aim of the present study is to identify a set of EEG indicators that can be used for the objective assessment of cognitive workload in a multitasking setting and as a foundational step toward a human-autonomy augmented cognition system.

Methods: The participants' perceived workload was modulated during a teleoperation task involving an unmanned aerial vehicle (UAV) shepherding a swarm of unmanned ground vehicles (UGVs). Three sources of data were recorded from sixteen participants (n = 16): heart rate (HR), EEG, and subjective indicators of the perceived workload using the Air Traffic Workload Input Technique (ATWIT).

Results: The HR data predicted the scores from ATWIT. Nineteen common EEG features offered a discriminatory power of the four workload setups with high classification accuracy (82.23%), exhibiting a higher sensitivity than ATWIT and HR.

Conclusion: The identified set of features represents EEG indicators for the objective assessment of cognitive workload across subjects. These common indicators could be used for augmented intelligence in human-autonomy teaming scenarios, and form the basis for our work on designing a closed-loop augmented cognition system for human-swarm teaming.

Prosocial Virtual Reality, Empathy, and EEG Measures: A Pilot Study Aimed at Monitoring Emotional Processes in Intergroup Helping Behaviors

During a non-invasive procedure, participants both helped and helped by a confederate with features that create social distance (membership in an ethnic outgroup or another social group). For this purpose, we created a set of virtual scenarios in which the confederate’s ethnicity (white vs. black) and appearance (business man vs. beggar, with casual dress as a control condition) were crossed. The study aimed to explore how the emotional reactions of participants changed according to the confederate’s status signals as well as signals that they belong to the same or a different ethnic group. Participants’ alertness, calmness, and engagement were monitored using electroencephalogram (EEG) during the original virtual reality (VR) video sessions. Participants’ distress and empathy when exposed to helping interactions were self-assessed after the VR video sessions. The results pointed out that, irrespective of whether they helped the confederate or were helped by him/her, white participants showed higher levels of alertness when exposed to helping interactions involving a white beggar or a black businessman, and their emotional calmness and engagement were higher when interacting with a black beggar or a white businessman. The results for self-assessed distress and empathy followed the same tendency, indicating how physiological and self-assessed measures can both contribute to a better understanding of the emotional processes in virtual intergroup helping situations. Based on the presented results, the methodological and practical implications of VR in terms of enhancing self-reflective capacities in intergroup helping processes are discussed.

Analysis of Decision-Making Process Using Methods of Quantitative Electroencephalography and Machine Learning Tools

The electroencephalographic activity of particular brain areas during the decision making process is still little known. This paper presents results of experiments on the group of 30 patients with a wide range of psychiatric disorders and 41 members of the control group. All subjects were performing the Iowa Gambling Task that is often used for decision process investigations. The electroencephalographical activity of participants was recorded using the dense array amplifier. The most frequently active Brodmann Areas were estimated by means of the photogrammetry techniques and source localization algorithms. The analysis was conducted in the full frequency as well as in alpha, beta, gamma, delta, and theta bands. Next the mean electric charge flowing through each of the most frequently active areas and for each frequency band was calculated. The comparison of the results obtained for the subjects and the control groups is presented. The difference in activity of the selected Brodmann Areas can be observed in all variants of the task. The hyperactivity of amygdala is found in both the patients and the control group. It is noted that the somatosensory association cortex, dorsolateral prefrontal cortex, and primary visual cortex play an important role in the decision-making process as well. Some of our results confirm the previous findings in the fMRI experiments. In addition, the results of the electroencephalographic analysis in the broadband as well as in specific frequency bands were used as inputs to several machine learning classifiers built in Azure Machine Learning environment. Comparison of classifiers' efficiency is presented to some extent and finding the most effective classifier may be important for planning research strategy toward finding decision-making biomarkers in cortical activity for both healthy people and those suffering from psychiatric disorders.

AttentivU: An EEG-Based Closed-Loop Biofeedback System for Real-Time Monitoring and Improvement of Engagement for Personalized Learning

Information about a person’s engagement and attention might be a valuable asset in many settings including work situations, driving, and learning environments. To this end, we propose the first prototype of a device called AttentivU—a system that uses a wearable system which consists of two main components. Component 1 is represented by an EEG headband used to measure the engagement of a person in real-time. Component 2 is a scarf, which provides subtle, haptic feedback (vibrations) in real-time when the drop in engagement is detected. We tested AttentivU in two separate studies with 48 adults. The participants were engaged in a learning scenario of either watching three video lectures on different subjects or participating in a set of three face-to-face lectures with a professor. There were three conditions administrated during both studies: (1) biofeedback, meaning the scarf (component 2 of the system) was vibrating each time the EEG headband detected a drop in engagement; (2) random feedback, where the vibrations did not correlate or depend on the engagement level detected by the system, and (3) no feedback, when no vibrations were administered. The results show that the biofeedback condition redirected the engagement of the participants to the task at hand and improved their performance on comprehension tests.

Toward a Hybrid Passive BCI for the Modulation of Sustained Attention Using EEG and fNIRS

We report results of a study that utilizes a BCI to drive an interactive interface countermeasure that allows users to self-regulate sustained attention while performing an ecologically valid, long-duration business logistics task. An engagement index derived from EEG signals was used to drive the BCI while fNIRS measured hemodynamic activity for the duration of the task. Participants (n = 30) were split into three groups (1) no countermeasures (NOCM), (2) continuous countermeasures (CCM), and (3) event synchronized, level-dependent countermeasures (ECM). We hypothesized that the ability to self-regulate sustained attention through a neurofeedback mechanism would result in greater task engagement, decreased error rate and improved task performance. Data were analyzed by wavelet coherence analysis, statistical analysis, performance metrics and self-assessed cognitive workload via RAW-TLX. We found that when the BCI was used to deliver continuous interface countermeasures (CCM), task performance was moderately enhanced in terms of total 14,785 (σ = 423) and estimated missed sales 7.46% (σ = 1.76) when compared to the NOCM 14,529 (σ = 510), 9.79% (σ = 2.75), and the ECM 14,180 (σ = 875), 9.62% (σ = 4.91) groups. An "actions per minute" (APM) metric was used to determine interface interaction activity which showed that overall the CCM and ECM groups had a higher APM of 3.460 (SE = 0.140) and 3.317 (SE = 0.139) respectively when compared with the NOCM group 2.65 (SE = 0.097). Statistical analysis showed a significant difference between ECM - NOCM and CCM - NOCM (p < 0.001) groups, but no significant difference between the ECM - CCM groups. Analysis of the RAW-TLX scores showed that the CCM group had lowest total score 7.27 (σ = 3.1) when compared with the ECM 9.7 (σ = 3.3) and NOCM 9.2 (σ = 3.4) groups. No statistical difference was found between the RAW-TLX or the subscales, except for self-perceived performance (p < 0.028) comparing the CCM and ECM groups. The results suggest that providing a means to self-regulate sustained attention has the potential to keep operators engaged over long periods, and moderately increase on-task performance while decreasing on-task error.

Predicting Premature Video Skipping and Viewer Interest from EEG Recordings

Brain–computer interfacing has enjoyed growing attention, not only due to the stunning demonstrations with severely disabled patients, but also the advent of economically viable solutions in areas such as neuromarketing, mental state monitoring, and future human–machine interaction. An interesting case, at least for neuromarketers, is to monitor the customer’s mental state in response to watching a commercial. In this paper, as a novelty, we propose a method to predict from electroencephalography (EEG) recordings whether individuals decide to skip watching a video trailer. Based on multiscale sample entropy and signal power, indices were computed that gauge the viewer’s engagement and emotional affect. We then trained a support vector machine (SVM), a k-nearest neighbor (kNN), and a random forest (RF) classifier to predict whether the viewer declares interest in watching the video and whether he/she decides to skip it prematurely. Our model achieved an average single-subject classification accuracy of 75.803% for skipping and 73.3% for viewer interest for the SVM, 82.223% for skipping and 78.333% for viewer interest for the kNN, and 80.003% for skipping and 75.555% for interest for the RF. We conclude that EEG can provide indications of viewer interest and skipping behavior and provide directions for future research.

Brain-Computer Interface-Based Adaptive Automation to Prevent Out-Of-The-Loop Phenomenon in Air Traffic Controllers Dealing With Highly Automated Systems

Increasing the level of automation in air traffic management is seen as a measure to increase the performance of the service to satisfy the predicted future demand. This is expected to result in new roles for the human operator: he will mainly monitor highly automated systems and seldom intervene. Therefore, air traffic controllers (ATCos) would often work in a supervisory or control mode rather than in a direct operating mode. However, it has been demonstrated how human operators in such a role are affected by human performance issues, known as Out-Of-The-Loop (OOTL) phenomenon, consisting in lack of attention, loss of situational awareness and de-skilling. A countermeasure to this phenomenon has been identified in the adaptive automation (AA), i.e., a system able to allocate the operative tasks to the machine or to the operator depending on their needs. In this context, psychophysiological measures have been highlighted as powerful tool to provide a reliable, unobtrusive and real-time assessment of the ATCo's mental state to be used as control logic for AA-based systems. In this paper, it is presented the so-called "Vigilance and Attention Controller", a system based on electroencephalography (EEG) and eye-tracking (ET) techniques, aimed to assess in real time the vigilance level of an ATCo dealing with a highly automated human-machine interface and to use this measure to adapt the level of automation of the interface itself. The system has been tested on 14 professional ATCos performing two highly realistic scenarios, one with the system disabled and one with the system enabled. The results confirmed that (i) long high automated tasks induce vigilance decreasing and OOTL-related phenomena; (ii) EEG measures are sensitive to these kinds of mental impairments; and (iii) AA was able to counteract this negative effect by keeping the ATCo more involved within the operative task. The results were confirmed by EEG and ET measures as well as by performance and subjective ones, providing a clear example of potential applications and related benefits of AA.

How Neurophysiological Measures Can be Used to Enhance the Evaluation of Remote Tower Solutions

New solutions in operational environments are often, among objective measurements, evaluated by using subjective assessment and judgment from experts. Anyhow, it has been demonstrated that subjective measures suffer from poor resolution due to a high intra and inter-operator variability. Also, performance measures, if available, could provide just partial information, since an operator could achieve the same performance but experiencing a different workload. In this study, we aimed to demonstrate: (i) the higher resolution of neurophysiological measures in comparison to subjective ones; and (ii) how the simultaneous employment of neurophysiological measures and behavioral ones could allow a holistic assessment of operational tools. In this regard, we tested the effectiveness of an electroencephalography (EEG)-based neurophysiological index (W_EEG index) in comparing two different solutions (i.e., Normal and Augmented) in terms of experienced workload. In this regard, 16 professional air traffic controllers (ATCOs) have been asked to perform two operational scenarios. Galvanic Skin Response (GSR) has also been recorded to evaluate the level of arousal (i.e., operator involvement) during the two scenarios execution. NASA-TLX questionnaire has been used to evaluate the perceived workload, and an expert was asked to assess performance achieved by the ATCOs. Finally, reaction times on specific operational events relevant for the assessment of the two solutions, have also been collected. Results highlighted that the Augmented solution induced a local increase in subjects performance (Reaction times). At the same time, this solution induced an increase in the workload experienced by the participants (W_EEG). Anyhow, this increase is still acceptable, since it did not negatively impact the performance and has to be intended only as a consequence of the higher engagement of the ATCOs. This behavioral effect is totally in line with physiological results obtained in terms of arousal (GSR), that increased during the scenario with augmentation. Subjective measures (NASA-TLX) did not highlight any significant variation in perceived workload. These results suggest that neurophysiological measure provide additional information than behavioral and subjective ones, even at a level of few seconds, and its employment during the pre-operational activities (e.g., design process) could allow a more holistic and accurate evaluation of new solutions.

Sensor Networks for Aerospace Human-Machine Systems

Intelligent automation and trusted autonomy are being introduced in aerospace cyber-physical systems to support diverse tasks including data processing, decision-making, information sharing and mission execution. Due to the increasing level of integration/collaboration between humans and automation in these tasks, the operational performance of closed-loop human-machine systems can be enhanced when the machine monitors the operator's cognitive states and adapts to them in order to maximise the effectiveness of the Human-Machine Interfaces and Interactions (HMI²). Technological developments have led to neurophysiological observations becoming a reliable methodology to evaluate the human operator's states using a variety of wearable and remote sensors. The adoption of sensor networks can be seen as an evolution of this approach, as there are notable advantages if these sensors collect and exchange data in real-time, while their operation is controlled remotely and synchronised. This paper discusses recent advances in sensor networks for aerospace cyber-physical systems, focusing on Cognitive HMI² (CHMI²) implementations. The key neurophysiological measurements used in this context and their relationship with the operator's cognitive states are discussed. Suitable data analysis techniques based on machine learning and statistical inference are also presented, as these techniques allow processing both neurophysiological and operational data to obtain accurate cognitive state estimations. Lastly, to support the development of sensor networks for CHMI² applications, the paper addresses the performance characterisation of various state-of-the-art sensors and the propagation of measurement uncertainties through a machine learning-based inference engine. Results show that a proper sensor selection and integration can support the implementation of effective human-machine systems for various challenging aerospace applications, including Air Traffic Management (ATM), commercial airliner Single-Pilot Operations (SIPO), one-to-many Unmanned Aircraft Systems (UAS), and space operations management.

Analyzing Passive BCI Signals to Control Adaptive Automation Devices

Brain computer interfaces are currently considered to greatly enhance assistive technologies and improve the experiences of people with special needs in the workplace. The proposed adaptive control model for smart offices provides a complete prototype that senses an environment's temperature and lighting and responds to users' feelings in terms of their comfort and engagement levels. The model comprises the following components: (a) sensors to sense the environment, including temperature and brightness sensors, and a headset that collects electroencephalogram (EEG) signals, which represent workers' comfort levels; (b) an application that analyzes workers' feelings regarding their willingness to adjust to a space based on an analysis of collected data and that determines workers' attention levels and, thus, engagement; and (c) actuators to adjust the temperature and/or lighting. This research implemented independent component analysis to remove eye movement artifacts from the EEG signals and used an engagement index to calculate engagement levels. This research is expected to add value to research on smart city infrastructures and on assistive technologies to increase productivity in smart offices.

EEG-Based Prediction of Cognitive Load in Intelligence Tests

Measuring and assessing the cognitive load associated with different tasks is crucial for many applications, from the design of instructional materials to monitoring the mental well-being of aircraft pilots. The goal of this paper is to utilize EEG to infer the cognitive workload of subjects during intelligence tests. We chose the well established advanced progressive matrices test, an ideal framework because it presents problems at increasing levels of difficulty and has been rigorously validated in past experiments. We train classic machine learning models using basic EEG measures as well as measures of network connectivity and signal complexity. Our findings demonstrate that cognitive load can be well predicted using these features, even for a low number of channels. We show that by creating an individually tuned neural network for each subject, we can improve prediction compared to a generic model and that such models are robust to decreasing the number of available channels as well.

Acoustic Neurofeedback Increases Beta ERD During Mental Rotation Task

The purpose of the present study was to identify the effect of acoustic neurofeedback on brain activity during consecutive stages of mental rotation of 3D objects. Given the fact that the process of mental rotation of objects is associated with desynchronisation of beta rhythm (beta ERD), it was expected that suppression in this band would be greater in the experimental group than in the controls. Thirty-three participants were randomly allocated to two groups performing the classic Shepard-Metzler mental rotation task (1971). The experimental group received auditory stimuli when the level of concentration fell below the threshold value determined separately for each participant based on the engagement index [β/(α + Θ)]. The level of concentration in the control group was not stimulated. Compared to the controls, the experimental group was found with greater beta-band suppression recorded above the left parietal cortex during the early stage and above the right parietal cortex during the late stage of mental rotation task. At the late stage of mental rotation, only the experimental group was found with differences in beta ERD related to varied degrees of the rotation angle and the control condition (zero angles, no rotation) recorded above the right parietal cortex and the central area of cerebral cortex. The present findings suggest that acoustic feedback might improve the process of mental rotation.

Texting while walking: An expensive switch cost

Texting while walking has been highlighted as a dangerous behavior that leads to impaired judgment and accidents. This impairment could be due to task switching which involves activation of the present task and the inhibition of the previous task. However, the relative contributions of these processes and their brain activity have not yet been studied. We addressed this gap by asking participants to discriminate the orientation of an oncoming human shape in a virtual environment while they were: i) walking on a treadmill, or ii) texting while walking on a treadmill. Participants' performance (i.e., correctly identifying if a walker would pass them to their left or right) and electroencephalography (EEG) data was collected. Unsurprisingly, we found that participants performed better while they were only walking than when texting while walking. However, we also found that the diminished performance is differently related to task set inhibition and task set activation in the two conditions. The alpha oscillations, which can be used as an index of task inhibition, have a significantly different relation to performance in the two conditions, the relation being negative when subjects are texting. This may indicate that the more inhibition is needed, the more the performance is affected by texting. To our knowledge, this is the first study to investigate the brain signature of task switching in texting while walking. This finding is the first step in identifying the source of impaired judgment in texting pedestrians and in finding viable solutions to reduce the risks.

Antismoking Campaigns' Perception and Gender Differences: A Comparison among EEG Indices

Human factors' aim is to understand and evaluate the interactions between people and tasks, technologies, and environment. Among human factors, it is possible then to include the subjective reaction to external stimuli, due to individual's characteristics and states of mind. These processes are also involved in the perception of antismoking public service announcements (PSAs), the main tool for governments to contrast the first cause of preventable deaths in the world: tobacco addiction. In the light of that, in the present article, it has been investigated through the comparison of different electroencephalographic (EEG) indices a typical item known to be able of influencing PSA perception, that is gender. In order to investigate the neurophysiological underpinnings of such different perception, we tested two PSAs: one with a female character and one with a male character. Furthermore, the experimental sample was divided into men and women, as well as smokers and nonsmokers. The employed EEG indices were the mental engagement (ME: the ratio between beta activity and the sum of alpha and theta activity); the approach/withdrawal (AW: the frontal alpha asymmetry in the alpha band); and the frontal theta activity and the spectral asymmetry index (SASI: the ratio between beta minus theta and beta plus theta). Results suggested that the ME and the AW presented an opposite trend, with smokers showing higher ME and lower AW than nonsmokers. The ME and the frontal theta also evidenced a statistically significant interaction between the kind of the PSA and the gender of the observers; specifically, women showed higher ME and frontal theta activity for the male character PSA. This study then supports the usefulness of the ME and frontal theta for purposes of PSAs targeting on the basis of gender issues and of the ME and the AW and for purposes of PSAs targeting on the basis of smoking habits.

Measuring mental workload using physiological measures: A systematic review

Technological advances have led to physiological measurement being increasingly used to measure and predict operator states. Mental workload (MWL) in particular has been characterised using a variety of physiological sensor data. This systematic review contributes a synthesis of the literature summarising key findings to assist practitioners to select measures for use in evaluation of MWL. We also describe limitations of the methods to assist selection when being deployed in applied or laboratory settings. We detail fifty-eight peer reviewed journal articles which present original data using physiological measures to include electrocardiographic, respiratory, dermal, blood pressure and ocular. Electroencephalographic measures have been included if they are presented with another measure to constrain scope. The literature reviewed covers a range of applied and experimental studies across various domains, safety-critical applications being highly represented in the sample of applied literature reviewed. We present a summary of the six measures and provide an evidence base which includes how to deploy each measure, and characteristics that can affect or preclude the use of a measure in research. Measures can be used to discriminate differences in MWL caused by task type, task load, and in some cases task difficulty. Varying ranges of sensitivity to sudden or gradual changes in taskload are also evident across the six measures. We conclude that there is no single measure that clearly discriminates mental workload but there is a growing empirical basis with which to inform both science and practice.

An Evaluation of EEG-based Metrics for Engagement Assessment of Distance Learners

Maintaining students' cognitive engagement in educational settings is crucial to their performance, though quantifying this mental state in real-time for distance learners has not been studied extensively in natural distance learning environments. We record electroencephalographic (EEG) data of students watching online lecture videos and use it to predict engagement rated by human annotators. An evaluation of prior EEG-based engagement metrics that utilize power spectral density (PSD) features is presented. We examine the predictive power of various supervised machine learning approaches with both subject-independent and individualized models when using simple PSD feature functions. Our results show that engagement metrics with few power band variables, including those proposed in prior research, do not produce predictions consistent with human observations. We quantify the performance disparity between cross-subject and per-subject models and demonstrate that individual differences in EEG patterns necessitate a more complex metric for educational engagement assessment in natural distance learning environments.

Mapping the Human Brain in Frequency Band Analysis of Brain Cortex Electroencephalographic Activity for Selected Psychiatric Disorders

There are still no good quantitative methods to be applied in psychiatric diagnosis. The interview is still the main and most important tool in the psychiatrist work. This paper presents the results of electroencephalographic research with the subjects of a group of 30 patients with psychiatric disorders compared to the control group of healthy volunteers. All subjects were solving working memory task. The digit-span working memory task test was chosen as one of the most popular tasks given to subjects with cognitive dysfunctions, especially for the patients with panic disorders, depression (including the depressive phase of bipolar disorder), phobias, and schizophrenia. Having such cohort of patients some results for the subjects with insomnia and Asperger syndrome are also presented. The cortical activity of their brains was registered by the dense array EEG amplifier. Source localization using the photogrammetry station and the sLORETA algorithm was then performed in five EEG frequency bands. The most active Brodmann Areas are indicated. Methodology for mapping the brain and research protocol are presented. The first results indicate that the presented technique can be useful in finding psychiatric disorder neurophysiological biomarkers. The first attempts were made to associate hyperactivity of selected Brodmann Areas with particular disorders.

Dynamic Threshold Selection for a Biocybernetic Loop in an Adaptive Video Game Context

Passive Brain-Computer interfaces (pBCIs) are a human-computer communication tool where the computer can detect from neurophysiological signals the current mental or emotional state of the user. The system can then adjust itself to guide the user toward a desired state. One challenge facing developers of pBCIs is that the system's parameters are generally set at the onset of the interaction and remain stable throughout, not adapting to potential changes over time such as fatigue. The goal of this paper is to investigate the improvement of pBCIs with settings adjusted according to the information provided by a second neurophysiological signal. With the use of a second signal, making the system a hybrid pBCI, those parameters can be continuously adjusted with dynamic thresholding to respond to variations such as fatigue or learning. In this experiment, we hypothesize that the adaptive system with dynamic thresholding will improve perceived game experience and objective game performance compared to two other conditions: an adaptive system with single primary signal biocybernetic loop and a control non-adaptive game. A within-subject experiment was conducted with 16 participants using three versions of the game Tetris. Each participant plays 15 min of Tetris under three experimental conditions. The control condition is the traditional game of Tetris with a progressive increase in speed. The second condition is a cognitive load only biocybernetic loop with the parameters presented in Ewing et al. (2016). The third condition is our proposed biocybernetic loop using dynamic threshold selection. Electroencephalography was used as the primary signal and automatic facial expression analysis as the secondary signal. Our results show that, contrary to our expectations, the adaptive systems did not improve the participants' experience as participants had more negative affect from the BCI conditions than in the control condition. We endeavored to develop a system that improved upon the authentic version of the Tetris game, however, our proposed adaptive system neither improved players' perceived experience, nor their objective performance. Nevertheless, this experience can inform developers of hybrid passive BCIs on a novel way to employ various neurophysiological features simultaneously.

Mental State Assessment and Validation Using Personalized Physiological Biometrics

Mental state monitoring is a critical component of current and future human-machine interfaces, including semi-autonomous driving and flying, air traffic control, decision aids, training systems, and will soon be integrated into ubiquitous products like cell phones and laptops. Current mental state assessment approaches supply quantitative measures, but their only frame of reference is generic population-level ranges. What is needed are physiological biometrics that are validated in the context of task performance of individuals. Using curated intake experiments, we are able to generate personalized models of three key biometrics as useful indicators of mental state; namely, mental fatigue, stress, and attention. We demonstrate improvements to existing approaches through the introduction of new features. Furthermore, addressing the current limitations in assessing the efficacy of biometrics for individual subjects, we propose and employ a multi-level validation scheme for the biometric models by means of k-fold cross-validation for discrete classification and regression testing for continuous prediction. The paper not only provides a unified pipeline for extracting a comprehensive mental state evaluation from a parsimonious set of sensors (only EEG and ECG), but also demonstrates the use of validation techniques in the absence of empirical data. Furthermore, as an example of the application of these models to novel situations, we evaluate the significance of correlations of personalized biometrics to the dynamic fluctuations of accuracy and reaction time on an unrelated threat detection task using a permutation test. Our results provide a path toward integrating biometrics into augmented human-machine interfaces in a judicious way that can help to maximize task performance.

New Protocol for Quantitative Analysis of Brain Cortex Electroencephalographic Activity in Patients With Psychiatric Disorders

The interview is still the main and most important tool in psychiatrist's work. The neuroimaging methods such as CT or MRI are widely used in other fields of medicine, for instance neurology. However, psychiatry lacks effective quantitative methods to support of diagnosis. A novel neuroinformatic approach to help clinical patients by means of electroencephalographic technology in order to build foundations for finding neurophysiological biomarkers of psychiatric disorders is proposed. A cohort of 30 right-handed patients (21 males, 9 females) with psychiatric disorders (mainly with panic and anxiety disorder, Asperger syndrome as well as with phobic anxiety disorders, schizophrenia, bipolar affective disorder, obsessive-compulsive disorder, nonorganic hypersomnia, and moderate depressive episode) were examined using the dense array EEG amplifier in the P300 experiment. The results were compared with the control group of 30 healthy, right-handed male volunteers. The quantitative analysis of cortical activity was conducted using the sLORETA source localization algorithm. The most active Brodmann Areas were pointed out and a new quantitative observable of electrical charge flowing through the selected Brodmann Area is proposed. The precise methodology and research protocol for collecting EEG data as well as the roadmap of future investigations in this area are presented. The essential result of this study is the idea proven by the initial results of our experiments that it is possible to determine quantitatively biomarkers of particular psychiatric disorders in order to support the process of diagnosis and hopefully choose most appropriate medical treatment later.

Dual Frequency Head Maps: A New Method for Indexing Mental Workload Continuously during Execution of Cognitive Tasks

One goal of advanced information and communication technology is to simplify work. However, there is growing consensus regarding the negative consequences of inappropriate workload on employee's health and the safety of persons. In order to develop a method for continuous mental workload monitoring, we implemented a task battery consisting of cognitive tasks with diverse levels of complexity and difficulty. We conducted experiments and registered the electroencephalogram (EEG), performance data, and the NASA-TLX questionnaire from 54 people. Analysis of the EEG spectra demonstrates an increase of the frontal theta band power and a decrease of the parietal alpha band power, both under increasing task difficulty level. Based on these findings we implemented a new method for monitoring mental workload, the so-called Dual Frequency Head Maps (DFHM) that are classified by support vectors machines (SVMs) in three different workload levels. The results are in accordance with the expected difficulty levels arising from the requirements of the tasks on the executive functions. Furthermore, this article includes an empirical validation of the new method on a secondary subset with new subjects and one additional new task without any adjustment of the classifiers. Hence, the main advantage of the proposed method compared with the existing solutions is that it provides an automatic, continuous classification of the mental workload state without any need for retraining the classifier—neither for new subjects nor for new tasks. The continuous workload monitoring can help ensure good working conditions, maintain a good level of performance, and simultaneously preserve a good state of health.

Exploring Audience Response in Performing Arts with a Brain-Adaptive Digital Performance System

Audience response is an important indicator of the quality of performing arts. Psychophysiological measurements enable researchers to perceive and understand audience response by collecting their bio-signals during a live performance. However, how the audience respond and how the performance is affected by these responses are the key elements but are hard to implement. To address this issue, we designed a brain-computer interactive system called Brain-Adaptive Digital Performance ( BADP ) for the measurement and analysis of audience engagement level through an interactive three-dimensional virtual theater. The BADP system monitors audience engagement in real time using electroencephalography (EEG) measurement and tries to improve it by applying content-related performing cues when the engagement level decreased.

In this article, we generate EEG-based engagement level and build thresholds to determine the decrease and re-engage moments. In the experiment, we simulated two types of theatre performance to provide participants a high-fidelity virtual environment using the BADP system. We also create content-related performing cues for each performance under three different conditions. The results of these evaluations show that our algorithm could accurately detect the engagement status and the performing cues have a positive impact on regaining audience engagement across different performance types. Our findings open new perspectives in audience-based theatre performance design.

EEG indices correlate with sustained attention performance in patients affected by diffuse axonal injury

The aim of this study is to assess the ability of EEG-based indices in providing relevant information about cognitive engagement level during the execution of a clinical sustained attention (SA) test in healthy volunteers and DAI (diffused axonal injury)-affected patients. We computed three continuous power-based engagement indices (P _β /P _α , 1/P _α , and P _β / (P _α + P _θ )) from EEG recordings in a control group (n = 7) and seven DAI-affected patients executing a 10-min Conners' "not-X" continuous performance test (CPT). A correlation analysis was performed in order to investigate the existence of relations between the EEG metrics and behavioral parameters in both the populations. P _β /P _α and 1/P _α indices were found to be correlated with reaction times in both groups while P _β / (P _α + P _θ ) and P _β /P _α also correlated with the errors rate for DAI patients. In line with previous studies, time course fluctuations revealed a first strong decrease of attention after 2 min from the beginning of the test and a final fading at the end. Our results provide evidence that EEG-derived indices extraction and evaluation during SA tasks are helpful in the assessment of attention level in healthy subjects and DAI patients, offering motivations for including EEG monitoring in cognitive rehabilitation practice. Graphical abstract Three EEG-derived indices were computed from four electrodes montages in a population of seven healthy volunteers and a group of seven DAI-affected patients. Results show a significant correlation between the time course of the indices and behavioral parameters, thus demonstrating their usefulness in monitoring mental engagement level during a sustained attention task.