Human Factors and Neurophysiological Metrics in Air Traffic Control: A Critical Review

Enhancing Aviation Safety through AI-Driven Mental Health Management for Pilots and Air Traffic Controllers

This article provides an overview of the mental health challenges faced by pilots and air traffic controllers (ATCs), whose stressful professional lives may negatively impact global flight safety and security. The adverse effects of mental health disorders on their flight performance pose a particular safety risk, especially in sudden unexpected startle situations. Therefore, the early detection, prediction and prevention of mental health deterioration in pilots and ATCs, particularly among those at high risk, are crucial to minimize potential air crash incidents caused by human factors. Recent research in artificial intelligence (AI) demonstrates the potential of machine and deep learning, edge and cloud computing, virtual reality and wearable multimodal physiological sensors for monitoring and predicting mental health disorders. Longitudinal monitoring and analysis of pilots' and ATCs physiological, cognitive and behavioral states could help predict individuals at risk of undisclosed or emerging mental health disorders. Utilizing AI tools and methodologies to identify and select these individuals for preventive mental health training and interventions could be a promising and effective approach to preventing potential air crash accidents attributed to human factors and related mental health problems. Based on these insights, the article advocates for the design of a multidisciplinary mental healthcare ecosystem in modern aviation using AI tools and technologies, to foster more efficient and effective mental health management, thereby enhancing flight safety and security standards. This proposed ecosystem requires the collaboration of multidisciplinary experts, including psychologists, neuroscientists, physiologists, psychiatrists, etc. to address these challenges in modern aviation.

Neurophysiological Assessment of An Innovative Maritime Safety System in Terms of Ship Operators' Mental Workload, Stress, and Attention in the Full Mission Bridge Simulator

The current industrial environment relies heavily on maritime transportation. Despite the continuous technological advances for the development of innovative safety software and hardware systems, there is a consistent gap in the scientific literature regarding the objective evaluation of the performance of maritime operators. The human factor is profoundly affected by changes in human performance or psychological state. The difficulty lies in the fact that the technology, tools, and protocols for investigating human performance are not fully mature or suitable for experimental investigation. The present research aims to integrate these two concepts by (i) objectively characterizing the psychological state of mariners, i.e., mental workload, stress, and attention, through their electroencephalographic (EEG) signal analysis, and (ii) validating an innovative safety framework countermeasure, defined as Human Risk-Informed Design (HURID), through the aforementioned neurophysiological approach. The proposed study involved 26 mariners within a high-fidelity bridge simulator while encountering collision risk in congested waters with and without the HURID. Subjective, behavioral, and neurophysiological data, i.e., EEG, were collected throughout the experimental activities. The results showed that the participants experienced a statistically significant higher mental workload and stress while performing the maritime activities without the HURID, while their attention level was statistically lower compared to the condition in which they performed the experiments with the HURID (all p < 0.05). Therefore, the presented study confirmed the effectiveness of the HURID during maritime operations in critical scenarios and led the way to extend the neurophysiological evaluation of the HFs of maritime operators during the performance of critical and/or standard shipboard tasks.

Detection of Pilot's Mental Workload Using a Wireless EEG Headset in Airfield Traffic Pattern Tasks

Elevated mental workload (MWL) experienced by pilots can result in increased reaction times or incorrect actions, potentially compromising flight safety. This study aims to develop a functional system to assist administrators in identifying and detecting pilots' real-time MWL and evaluate its effectiveness using designed airfield traffic pattern tasks within a realistic flight simulator. The perceived MWL in various situations was assessed and labeled using NASA Task Load Index (NASA-TLX) scores. Physiological features were then extracted using a fast Fourier transformation with 2-s sliding time windows. Feature selection was conducted by comparing the results of the Kruskal-Wallis (K-W) test and Sequential Forward Floating Selection (SFFS). The results proved that the optimal input was all PSD features. Moreover, the study analyzed the effects of electroencephalography (EEG) features from distinct brain regions and PSD changes across different MWL levels to further assess the proposed system's performance. A 10-fold cross-validation was performed on six classifiers, and the optimal accuracy of 87.57% was attained using a multi-class K-Nearest Neighbor (KNN) classifier for classifying different MWL levels. The findings indicate that the wireless headset-based system is reliable and feasible. Consequently, numerous wireless EEG device-based systems can be developed for application in diverse real-driving scenarios. Additionally, the current system contributes to future research on actual flight conditions.

Differences Between Good, Moderate and Poor Performers of a Verbal Fluency Task under Blue Light Exposure: An SPA-fNIRS Study

Individuals have different performance levels for cognitive tasks. Are these performance levels reflected in physiological parameters? The aim of this study was to address this question by systemic physiology augmented functional near-infrared spectroscopy (SPA-fNIRS). We aimed to investigate whether different verbal fluency task (VFT) performances under blue light exposure were associated with different changes in cerebrovascular oxygenation and systemic physiological activity. The VFT performance of 32 healthy subjects (17 female, 15 male, age: 25.5 ± 4.3 years) was investigated under blue light exposure (120 lux). The VFT, which contained letter and category fluency tasks, lasted 9 min. There were rest periods without light exposure before and after the VFT for 8 min and 15 min, respectively. Based on their number of correct responses, subjects were classified into three groups, i.e., good, moderate, and poor performers. During the entire experiment, we simultaneously measured changes in cerebral and systemic physiological parameters using the SPA-fNIRS approach. We found that the better the subject's performance was, the smaller the task-evoked changes in cerebrovascular hemodynamics and oxygenation in the prefrontal cortex. Performance-dependent changes were also evident for skin conductance, arterial oxygen saturation and mean arterial pressure. This is the first VFT study that applies the comprehensive SPA-fNIRS approach to determine the relationship between task performance and changes in cerebral oxygenation and systemic physiology. Our study shows that these parameters are indeed related and the performance is reflected in the task-evoked cerebrovascular and systemic physiological changes.

Aviation and neurophysiology: A systematic review

This paper systematically reviews 20 years of publications (N = 54) on aviation and neurophysiology. The main goal is to provide an account of neurophysiological changes associated with flight training with the aim of identifying neurometrics indicative of pilot's flight training level and task relevant mental states, as well as to capture the current state-of-art of (neuro)ergonomic design and practice in flight training. We identified multiple candidate neurometrics of training progress and workload, such as frontal theta power, the EEG Engagement Index and the Cognitive Stability Index. Furthermore, we discovered that several types of classifiers could be used to accurately detect mental states, such as the detection of drowsiness and mental fatigue. The paper advances practical guidelines on terminology usage, simulator fidelity, and multimodality, as well as future research ideas including the potential of Virtual Reality flight simulations for training, and a brain-computer interface for flight training.

Task difficulty and physiological measures of mental workload in air traffic control: a scoping review

This study provides a systematic synthesis of empirical research on mental workload (MWL) in air traffic control (ATC). MWL is a key concept in research on innovative technologies, because the assessment of MWL is crucial to the evaluation of such technologies. Our specific focus was on physiological measures of MWL. The used search strategy identified 39 peer-reviewed publications that analysed ATC tasks, examined different levels of difficulty of the ATC task, and considered at least one physiological measure of MWL. Positive relations between measures of MWL and task difficulty were observed most frequently, indicating that the measures indeed allowed the assessment of MWL. The most commonly used physiological measures were brain measures (EEG and fNIR) and heart rate measures. The review revealed a need for more precise descriptions of crucial experimental parameters in order to permit a transition of the field towards more interactive and dynamic types of analysis. Practitioner summary: Research on innovative technology in air traffic control (ATC) depends on assessments of mental workload (MWL). We reviewed empirical research on MWL in ATC. Brain and heart measures often allow assessments of MWL. Better descriptions of experiments are needed to allow comparisons among studies and more dynamic and interactive analyses.

Evaluation of a New Lightweight EEG Technology for Translational Applications of Passive Brain-Computer Interfaces

Technologies like passive brain-computer interfaces (BCI) can enhance human-machine interaction. Anyhow, there are still shortcomings in terms of easiness of use, reliability, and generalizability that prevent passive-BCI from entering real-life situations. The current work aimed to technologically and methodologically design a new gel-free passive-BCI system for out-of-the-lab employment. The choice of the water-based electrodes and the design of a new lightweight headset met the need for easy-to-wear, comfortable, and highly acceptable technology. The proposed system showed high reliability in both laboratory and realistic settings, performing not significantly different from the gold standard based on gel electrodes. In both cases, the proposed system allowed effective discrimination (AUC > 0.9) between low and high levels of workload, vigilance, and stress even for high temporal resolution (<10 s). Finally, the generalizability of the proposed system has been tested through a cross-task calibration. The system calibrated with the data recorded during the laboratory tasks was able to discriminate the targeted human factors during the realistic task reaching AUC values higher than 0.8 at 40 s of temporal resolution in case of vigilance and workload, and 20 s of temporal resolution for the stress monitoring. These results pave the way for ecologic use of the system, where calibration data of the realistic task are difficult to obtain.

Real-time Pilot Crew's Mental Workload and Arousal Assessment During Simulated Flights for Training Evaluation: a Case Study

Training assessment is usually done by evaluating information derived from instructor's supervision related to the pilot's operational performance and behavior. However, this approach lacks objective measures, especially regarding the pilots' mental states while accomplishing the flight training tasks. The study therefore aimed at developing and testing a method for gathering and analyzing in real-time pilots' brain activity and skin conductance to improve the training evaluation. In this regard, Novice pilots' neurophysiological signals were acquired throughout multi-crew training sessions. The results demonstrated how the methodology proposed was able to endow real-time pilots' mental workload and arousal assessment for i) better evaluating training progress and operational behavior during the training session, and ii) for objectively comparing different training sessions.

Danger, high voltage! Using EEG and EOG measurements for cognitive overload detection in a simulated industrial context

Industrial settings will be characterized by far-reaching production automation brought about by advancements in robotics and artificial intelligence. As a consequence, human assembly workers will need to adapt quickly to new and more complex assembly procedures, which are most likely to increase cognitive workload, or potentially induce overload. Measurement and optimization protocols need to be developed in order to be able to monitor workers' cognitive load. Previous studies have used electroencephalographic (EEG, measuring brain activity) and electrooculographic (EOG, measuring eye movements) signals, using basic computer-based static tasks and without creating an experience of overload. In this study, EEG and EOG data was collected of 46 participants performing an ecologically valid assembly task while inducing three levels of cognitive load (low, high and overload). The lower individual alpha frequency (IAF) was identified as a promising marker for discriminating between different levels of cognitive load and overload.

Unravelling the Physiological Correlates of Mental Workload Variations in Tracking and Collision Prediction Tasks

Modern work environments have extensive interactions with technology and greater cognitive complexity of the tasks, which results in human operators experiencing increased mental workload. Air traffic control operators routinely work in such complex environments, and we designed tracking and collision prediction tasks to emulate their elementary tasks. The physiological response to the workload variations in these tasks was elucidated to untangle the impact of workload variations experienced by operators. Electroencephalogram (EEG), eye activity, and heart rate variability (HRV) data were recorded from 24 participants performing tracking and collision prediction tasks with three levels of difficulty. Our findings indicate that variations in task load in both these tasks are sensitively reflected in EEG, eye activity and HRV data. Multiple regression results also show that operators' performance in both tasks can be predicted using the corresponding EEG, eye activity and HRV data. The results also demonstrate that the brain dynamics during each of these tasks can be estimated from the corresponding eye activity, HRV and performance data. Furthermore, the markedly distinct neurometrics of workload variations in the tracking and collision prediction tasks indicate that neurometrics can provide insights on the type of mental workload. These findings have applicability to the design of future mental workload adaptive systems that integrate neurometrics in deciding not just "when" but also "what" to adapt. Our study provides compelling evidence in the viability of developing intelligent closed-loop mental workload adaptive systems that ensure efficiency and safety in complex work environments.

Validation of a Light EEG-Based Measure for Real-Time Stress Monitoring during Realistic Driving

Driver’s stress affects decision-making and the probability of risk occurrence, and it is therefore a key factor in road safety. This suggests the need for continuous stress monitoring. This work aims at validating a stress neurophysiological measure—a Neurometric—for out-of-the-lab use obtained from lightweight EEG relying on two wet sensors, in real-time, and without calibration. The Neurometric was tested during a multitasking experiment and validated with a realistic driving simulator. Twenty subjects participated in the experiment, and the resulting stress Neurometric was compared with the Random Forest (RF) model, calibrated by using EEG features and both intra-subject and cross-task approaches. The Neurometric was also compared with a measure based on skin conductance level (SCL), representing one of the physiological parameters investigated in the literature mostly correlated with stress variations. We found that during both multitasking and realistic driving experiments, the Neurometric was able to discriminate between low and high levels of stress with an average Area Under Curve (AUC) value higher than 0.9. Furthermore, the stress Neurometric showed higher AUC and stability than both the SCL measure and the RF calibrated with a cross-task approach. In conclusion, the Neurometric proposed in this work proved to be suitable for out-of-the-lab monitoring of stress levels.

Measuring the Domain Specificity of Workload Using EEG: Auditory and Visual Domains in Rotary-Wing Simulated Flight

OBJECTIVE

The overarching objective was to evaluate whether workload sensory-domain specificity could be identified through electroencephalogram (EEG) recordings during simulated rotary-wing operations.

BACKGROUND

Rotary-wing aviators experience workload from different sensory domains, although predominantly through auditory and visual domains. Development of real-time monitoring tools using psychophysiological indices, such as EEG recordings, could enable identification of aviator overload in real time.

METHOD

Two studies were completed, both of which recorded EEG, task performance, and self-report data. In Study 1, 16 individuals completed a basic auditory and a basic visual laboratory task where workload was manipulated. In Study 2, 23 Army aviators completed simulated aviation flights where workload was manipulated within auditory and visual sensory domains.

RESULTS

Results from Study 1 found differences in frontal alpha activity during the auditory task, and that alpha and beta activities were associated with perceived workload. Frontal theta activity was found to differ during the visual task while frontal alpha was associated with perceived workload. Study 2 found support for frontal beta activity and the ratio of beta to alpha + theta to differentiate level of workload within the auditory domain.

CONCLUSION

There is likely a role of frontal alpha and beta activities in response to workload manipulations within the auditory domain; however, this role becomes more equivocal when examined in a multifaceted flight scenario.

APPLICATION

Results from this study provide a basis for understanding changes in EEG activity when workload is manipulated in sensory domains that can be used in furthering the development of real-time monitoring tools.

Anytime collaborative brain-computer interfaces for enhancing perceptual group decision-making

In this paper we present, and test in two realistic environments, collaborative Brain-Computer Interfaces (cBCIs) that can significantly increase both the speed and the accuracy of perceptual group decision-making. The key distinguishing features of this work are: (1) our cBCIs combine behavioural, physiological and neural data in such a way as to be able to provide a group decision at any time after the quickest team member casts their vote, but the quality of a cBCI-assisted decision improves monotonically the longer the group decision can wait; (2) we apply our cBCIs to two realistic scenarios of military relevance (patrolling a dark corridor and manning an outpost at night where users need to identify any unidentified characters that appear) in which decisions are based on information conveyed through video feeds; and (3) our cBCIs exploit Event-Related Potentials (ERPs) elicited in brain activity by the appearance of potential threats but, uniquely, the appearance time is estimated automatically by the system (rather than being unrealistically provided to it). As a result of these elements, in the two test environments, groups assisted by our cBCIs make both more accurate and faster decisions than when individual decisions are integrated in more traditional manners.

Explanation of Machine-Learning Solutions in Air-Traffic Management

Advances in the trusted autonomy of air-traffic management (ATM) systems are currently being pursued to cope with the predicted growth in air-traffic densities in all classes of airspace. Highly automated ATM systems relying on artificial intelligence (AI) algorithms for anomaly detection, pattern identification, accurate inference, and optimal conflict resolution are technically feasible and demonstrably able to take on a wide variety of tasks currently accomplished by humans. However, the opaqueness and inexplicability of most intelligent algorithms restrict the usability of such technology. Consequently, AI-based ATM decision-support systems (DSS) are foreseen to integrate eXplainable AI (XAI) in order to increase interpretability and transparency of the system reasoning and, consequently, build the human operators’ trust in these systems. This research presents a viable solution to implement XAI in ATM DSS, providing explanations that can be appraised and analysed by the human air-traffic control operator (ATCO). The maturity of XAI approaches and their application in ATM operational risk prediction is investigated in this paper, which can support both existing ATM advisory services in uncontrolled airspace (Classes E and F) and also drive the inflation of avoidance volumes in emerging performance-driven autonomy concepts. In particular, aviation occurrences and meteorological databases are exploited to train a machine learning (ML)-based risk-prediction tool capable of real-time situation analysis and operational risk monitoring. The proposed approach is based on the XGBoost library, which is a gradient-boost decision tree algorithm for which post-hoc explanations are produced by SHapley Additive exPlanations (SHAP) and Local Interpretable Model-Agnostic Explanations (LIME). Results are presented and discussed, and considerations are made on the most promising strategies for evolving the human–machine interactions (HMI) to strengthen the mutual trust between ATCO and systems. The presented approach is not limited only to conventional applications but also suitable for UAS-traffic management (UTM) and other emerging applications.

Prediction of Task Performance From Physiological Features of Stress Resilience

In this paper, we investigate the potential of generic physiological features of stress resilience in predicting air traffic control (ATC) candidates' performance in a highly-stressful low-fidelity ATC simulator scenario. Stress resilience is highlighted as an important occupational factor that influences the performance and well-being of air traffic control officers (ATCO). Poor stress management, besides the lack of skills, can be a direct cause of poor performance under stress, both in the selection process of ATCOs and later in the workplace. 40 ATC candidates, within the final stages of their selection process, underwent a stimulation paradigm for elicitation and assessment of various generic task-unrelated physiological features, related to resting heart rate variability (HRV) and respiratory sinus arrhythmia (RSA), acoustic startle response (ASR) and the physiological allostatic response, which are all recognized as relevant psychophysiological markers of stress resilience. The multimodal approach included analysis of electrocardiography, electromyography, electrodermal activity and respiration. We make advances in computational methodology for assessment of physiological features of stress resilience, and investigate the predictive power of the obtained feature space in a binary classification problem: prediction of high- vs. low-performance on the developed ATC simulator. Our novel approach yields a relatively high 78.16% classification accuracy. These results are discussed in the context of prior work, while considering study limitations and proposing directions for future work.

Wearable Technologies for Mental Workload, Stress, and Emotional State Assessment during Working-Like Tasks: A Comparison with Laboratory Technologies

The capability of monitoring user’s performance represents a crucial aspect to improve safety and efficiency of several human-related activities. Human errors are indeed among the major causes of work-related accidents. Assessing human factors (HFs) could prevent these accidents through specific neurophysiological signals’ evaluation but laboratory sensors require highly-specialized operators and imply a certain grade of invasiveness which could negatively interfere with the worker’s activity. On the contrary, consumer wearables are characterized by their ease of use and their comfortability, other than being cheaper compared to laboratory technologies. Therefore, wearable sensors could represent an ideal substitute for laboratory technologies for a real-time assessment of human performances in ecological settings. The present study aimed at assessing the reliability and capability of consumer wearable devices (i.e., Empatica E4 and Muse 2) in discriminating specific mental states compared to laboratory equipment. The electrooculographic (EOG), electrodermal activity (EDA) and photoplethysmographic (PPG) signals were acquired from a group of 17 volunteers who took part to the experimental protocol in which different working scenarios were simulated to induce different levels of mental workload, stress, and emotional state. The results demonstrated that the parameters computed by the consumer wearable and laboratory sensors were positively and significantly correlated and exhibited the same evidences in terms of mental states discrimination.

An Introductory Tutorial on Brain–Computer Interfaces and Their Applications

The prospect and potentiality of interfacing minds with machines has long captured human imagination. Recent advances in biomedical engineering, computer science, and neuroscience are making brain–computer interfaces a reality, paving the way to restoring and potentially augmenting human physical and mental capabilities. Applications of brain–computer interfaces are being explored in applications as diverse as security, lie detection, alertness monitoring, gaming, education, art, and human cognition augmentation. The present tutorial aims to survey the principal features and challenges of brain–computer interfaces (such as reliable acquisition of brain signals, filtering and processing of the acquired brainwaves, ethical and legal issues related to brain–computer interface (BCI), data privacy, and performance assessment) with special emphasis to biomedical engineering and automation engineering applications. The content of this paper is aimed at students, researchers, and practitioners to glimpse the multifaceted world of brain–computer interfacing.

Forefront Users' Experience Evaluation by Employing Together Virtual Reality and Electroencephalography: A Case Study on Cognitive Effects of Scents

Scents have the ability to affect peoples' mental states and task performance with to different extents. It has been widely demonstrated that the lemon scent, included in most all-purpose cleaners, elicits stimulation and activation, while the lavender scent elicits relaxation and sedative effects. The present study aimed at investigating and fostering a novel approach to evaluate users' experience with respect to scents' effects through the joint employment of Virtual Reality and users' neurophysiological monitoring, in particular Electroencephalography. In particular, this study, involving 42 participants, aimed to compare the effects of lemon and lavender scents on the deployment of cognitive resources during a daily life experience consisting in a train journey carried out in virtual reality. Our findings showed a significant higher request of cognitive resources during the processing of an informative message for subjects exposed to the lavender scent with respect to the lemon exposure. No differences were found between lemon and lavender conditions on the self-reported items of pleasantness and involvement; as this study demonstrated, the employment of the lavender scent preserves the quality of the customer experience to the same extent as the more widely used lemon scent.

Beyond Technologies of Electroencephalography-Based Brain-Computer Interfaces: A Systematic Review From Commercial and Ethical Aspects

The deployment of electroencephalographic techniques for commercial applications has undergone a rapid growth in recent decades. As they continue to expand in the consumer markets as suitable techniques for monitoring the brain activity, their transformative potential necessitates equally significant ethical inquiries. One of the main questions, which arises then when evaluating these kinds of applications, is whether they should be aligned or not with the main ethical concerns reported by scholars and experts. Thus, the present work attempts to unify these disciplines of knowledge by performing a comprehensive scan of the major electroencephalographic market applications as well as their most relevant ethical concerns arising from the existing literature. In this literature review, different databases were consulted, which presented conceptual and empirical discussions and findings about commercial and ethical aspects of electroencephalography. Subsequently, the content was extracted from the articles and the main conclusions were presented. Finally, an external assessment of the outcomes was conducted in consultation with an expert panel in some of the topic areas such as biomedical engineering, biomechatronics, and neuroscience. The ultimate purpose of this review is to provide a genuine insight into the cutting-edge practical attempts at electroencephalography. By the same token, it seeks to highlight the overlap between the market needs and the ethical standards that should govern the deployment of electroencephalographic consumer-grade solutions, providing a practical approach that overcomes the engineering myopia of certain ethical discussions.

A multimodal and signals fusion approach for assessing the impact of stressful events on Air Traffic Controllers

Stress is a word used to describe human reactions to emotionally, cognitively and physically challenging experiences. A hallmark of the stress response is the activation of the autonomic nervous system, resulting in the "fight-freeze-flight" response to a threat from a dangerous situation. Consequently, the capability to objectively assess and track a controller's stress level while dealing with air traffic control (ATC) activities would make it possible to better tailor the work shift and maintain high safety levels, as well as to preserve the operator's health. In this regard, sixteen controllers were asked to perform a realistic air traffic management (ATM) simulation during which subjective data (i.e. stress perception) and neurophysiological data (i.e. brain activity, heart rate, and galvanic skin response) were collected with the aim of accurately characterising the controller's stress level experienced in the various experimental conditions. In addition, external supervisors regularly evaluated the controllers in terms of manifested stress, safety, and efficiency throughout the ATM scenario. The results demonstrated 1) how the stressful events caused both supervisors and controllers to underestimate the experienced stress level, 2) the advantage of taking into account both cognitive and hormonal processes in order to define a reliable stress index, and 3) the importance of the points in time at which stress is measured owing to the potential transient effect once the stressful events have ceased.

Indexing Mental Workload During Simulated Air Traffic Control Tasks by Means of Dual Frequency Head Maps

In our digitized society, advanced information and communication technology and highly interactive work environments impose high demands on cognitive capacity. Optimal workload conditions are important for assuring employee's health and safety of other persons. This is particularly relevant in safety-critical occupations, such as air traffic control. For measuring mental workload using the EEG, we have developed the method of Dual Frequency Head Maps (DFHM). The method was tested and validated already under laboratory conditions. However, validation of the method regarding reliability and reproducibility of results under realistic settings and real world scenarios was still required. In our study, we examined 21 air traffic controllers during arrival management tasks. Mental workload variations were achieved by simulation scenarios with different number of aircraft and the occurrence of a priority-flight request as an exceptional event. The workload was assessed using the EEG-based DFHM-workload index and instantaneous self-assessment questionnaire. The DFHM-workload index gave stable results with highly significant correlations between scenarios with similar traffic-load conditions (r between 0.671 and 0.809, p ≤ 0.001). For subjects reporting that they experienced workload variation between the different scenarios, the DFHM-workload index yielded significant differences between traffic-load levels and priority-flight request conditions. For subjects who did not report to experience workload variations between the scenarios, the DFHM-workload index did not yield any significant differences for any of the factors. We currently conclude that the DFHM-workload index reveals potential for applications outside the laboratory and yields stable results without retraining of the classifiers neither regarding new subjects nor new tasks.

Neurophysiological Vigilance Characterisation and Assessment: Laboratory and Realistic Validations Involving Professional Air Traffic Controllers

Vigilance degradation usually causes significant performance decrement. It is also considered the major factor causing the out-of-the-loop phenomenon (OOTL) occurrence. OOTL is strongly related to a high level of automation in operative contexts such as the Air Traffic Management (ATM), and it could lead to a negative impact on the Air Traffic Controllers’ (ATCOs) engagement. As a consequence, being able to monitor the ATCOs’ vigilance would be very important to prevent risky situations. In this context, the present study aimed to characterise and assess the vigilance level by using electroencephalographic (EEG) measures. The first study, involving 13 participants in laboratory settings allowed to find out the neurophysiological features mostly related to vigilance decrements. Those results were also confirmed under realistic ATM settings recruiting 10 professional ATCOs. The results demonstrated that (i) there was a significant performance decrement related to vigilance reduction; (ii) there were no substantial differences between the identified neurophysiological features in controlled and ecological settings, and the EEG-channel configuration defined in laboratory was able to discriminate and classify vigilance changes in ATCOs’ vigilance with high accuracy (up to 84%); (iii) the derived two EEG-channel configuration was able to assess vigilance variations reporting only slight accuracy reduction.

EEG-Based Multi-Modal Emotion Recognition using Bag of Deep Features: An Optimal Feature Selection Approach

Much attention has been paid to the recognition of human emotions with the help of electroencephalogram (EEG) signals based on machine learning technology. Recognizing emotions is a challenging task due to the non-linear property of the EEG signal. This paper presents an advanced signal processing method using the deep neural network (DNN) for emotion recognition based on EEG signals. The spectral and temporal components of the raw EEG signal are first retained in the 2D Spectrogram before the extraction of features. The pre-trained AlexNet model is used to extract the raw features from the 2D Spectrogram for each channel. To reduce the feature dimensionality, spatial, and temporal based, bag of deep features (BoDF) model is proposed. A series of vocabularies consisting of 10 cluster centers of each class is calculated using the k-means cluster algorithm. Lastly, the emotion of each subject is represented using the histogram of the vocabulary set collected from the raw-feature of a single channel. Features extracted from the proposed BoDF model have considerably smaller dimensions. The proposed model achieves better classification accuracy compared to the recently reported work when validated on SJTU SEED and DEAP data sets. For optimal classification performance, we use a support vector machine (SVM) and k-nearest neighbor (k-NN) to classify the extracted features for the different emotional states of the two data sets. The BoDF model achieves 93.8% accuracy in the SEED data set and 77.4% accuracy in the DEAP data set, which is more accurate compared to other state-of-the-art methods of human emotion recognition.

Consumer Behaviour through the Eyes of Neurophysiological Measures: State-of-the-Art and Future Trends

The new technological advances achieved during the last decade allowed the scientific community to investigate and employ neurophysiological measures not only for research purposes but also for the study of human behaviour in real and daily life situations. The aim of this review is to understand how and whether neuroscientific technologies can be effectively employed to better understand the human behaviour in real decision-making contexts. To do so, firstly, we will describe the historical development of neuromarketing and its main applications in assessing the sensory perceptions of some marketing and advertising stimuli. Then, we will describe the main neuroscientific tools available for such kind of investigations (e.g., measuring the cerebral electrical or hemodynamic activity, the eye movements, and the psychometric responses). Also, this review will present different brain measurement techniques, along with their pros and cons, and the main cerebral indexes linked to the specific mental states of interest (used in most of the neuromarketing research). Such indexes have been supported by adequate validations from the scientific community and are largely employed in neuromarketing research. This review will also discuss a series of papers that present different neuromarketing applications, such us in-store choices and retail, services, pricing, brand perception, web usability, neuropolitics, evaluation of the food and wine taste, and aesthetic perception of artworks. Furthermore, this work will face the ethical issues arisen on the use of these tools for the evaluation of the human behaviour during decision-making tasks. In conclusion, the main challenges that neuromarketing is going to face, as well as future directions and possible scenarios that could be derived by the use of neuroscience in the marketing field, will be identified and discussed.

Brain-Computer Interface-Based Adaptive Automation to Prevent Out-Of-The-Loop Phenomenon in Air Traffic Controllers Dealing With Highly Automated Systems

Increasing the level of automation in air traffic management is seen as a measure to increase the performance of the service to satisfy the predicted future demand. This is expected to result in new roles for the human operator: he will mainly monitor highly automated systems and seldom intervene. Therefore, air traffic controllers (ATCos) would often work in a supervisory or control mode rather than in a direct operating mode. However, it has been demonstrated how human operators in such a role are affected by human performance issues, known as Out-Of-The-Loop (OOTL) phenomenon, consisting in lack of attention, loss of situational awareness and de-skilling. A countermeasure to this phenomenon has been identified in the adaptive automation (AA), i.e., a system able to allocate the operative tasks to the machine or to the operator depending on their needs. In this context, psychophysiological measures have been highlighted as powerful tool to provide a reliable, unobtrusive and real-time assessment of the ATCo's mental state to be used as control logic for AA-based systems. In this paper, it is presented the so-called "Vigilance and Attention Controller", a system based on electroencephalography (EEG) and eye-tracking (ET) techniques, aimed to assess in real time the vigilance level of an ATCo dealing with a highly automated human-machine interface and to use this measure to adapt the level of automation of the interface itself. The system has been tested on 14 professional ATCos performing two highly realistic scenarios, one with the system disabled and one with the system enabled. The results confirmed that (i) long high automated tasks induce vigilance decreasing and OOTL-related phenomena; (ii) EEG measures are sensitive to these kinds of mental impairments; and (iii) AA was able to counteract this negative effect by keeping the ATCo more involved within the operative task. The results were confirmed by EEG and ET measures as well as by performance and subjective ones, providing a clear example of potential applications and related benefits of AA.

How Neurophysiological Measures Can be Used to Enhance the Evaluation of Remote Tower Solutions

New solutions in operational environments are often, among objective measurements, evaluated by using subjective assessment and judgment from experts. Anyhow, it has been demonstrated that subjective measures suffer from poor resolution due to a high intra and inter-operator variability. Also, performance measures, if available, could provide just partial information, since an operator could achieve the same performance but experiencing a different workload. In this study, we aimed to demonstrate: (i) the higher resolution of neurophysiological measures in comparison to subjective ones; and (ii) how the simultaneous employment of neurophysiological measures and behavioral ones could allow a holistic assessment of operational tools. In this regard, we tested the effectiveness of an electroencephalography (EEG)-based neurophysiological index (W_EEG index) in comparing two different solutions (i.e., Normal and Augmented) in terms of experienced workload. In this regard, 16 professional air traffic controllers (ATCOs) have been asked to perform two operational scenarios. Galvanic Skin Response (GSR) has also been recorded to evaluate the level of arousal (i.e., operator involvement) during the two scenarios execution. NASA-TLX questionnaire has been used to evaluate the perceived workload, and an expert was asked to assess performance achieved by the ATCOs. Finally, reaction times on specific operational events relevant for the assessment of the two solutions, have also been collected. Results highlighted that the Augmented solution induced a local increase in subjects performance (Reaction times). At the same time, this solution induced an increase in the workload experienced by the participants (W_EEG). Anyhow, this increase is still acceptable, since it did not negatively impact the performance and has to be intended only as a consequence of the higher engagement of the ATCOs. This behavioral effect is totally in line with physiological results obtained in terms of arousal (GSR), that increased during the scenario with augmentation. Subjective measures (NASA-TLX) did not highlight any significant variation in perceived workload. These results suggest that neurophysiological measure provide additional information than behavioral and subjective ones, even at a level of few seconds, and its employment during the pre-operational activities (e.g., design process) could allow a more holistic and accurate evaluation of new solutions.

The effects of dynamic workload and experience on commercially available EEG cognitive state metrics in a high-fidelity air traffic control environment

The current study evaluated the validity of commercially available electroencephalography (EEG) cognitive state metrics of workload and engagement in differentially experienced air traffic control (ATC) students. EEG and pupil diameter recordings were collected from 47 ATC students (27 more experienced and 20 less experienced) during a high-fidelity, variable workload approach-control scenario. Scenario workload was manipulated by increasing the number of aircraft released and the presence of a divided attention task. Results showed that scenario performance significantly degraded with increased aircraft and the presence of the divided attention task. No scenario performance differences were found between experience groups. The EEG engagement metric significantly differed between experience groups, with less experienced controllers exhibiting higher engagement than more experienced controllers. The EEG workload metric and pupil diameter were sensitive to workload manipulations but did not differentiate experience groups. Commercially available EEG cognitive state metrics may be a viable tool for enhancing ATC training.

The Dry Revolution: Evaluation of Three Different EEG Dry Electrode Types in Terms of Signal Spectral Features, Mental States Classification and Usability

One century after the first recording of human electroencephalographic (EEG) signals, EEG has become one of the most used neuroimaging techniques. The medical devices industry is now able to produce small and reliable EEG systems, enabling a wide variety of applications also with no-clinical aims, providing a powerful tool to neuroscientific research. However, these systems still suffer from a critical limitation, consisting in the use of wet electrodes, that are uncomfortable and require expertise to install and time from the user. In this context, dozens of different concepts of EEG dry electrodes have been recently developed, and there is the common opinion that they are reaching traditional wet electrodes quality standards. However, although many papers have tried to validate them in terms of signal quality and usability, a comprehensive comparison of different dry electrode types from multiple points of view is still missing. The present work proposes a comparison of three different dry electrode types, selected among the main solutions at present, against wet electrodes, taking into account several aspects, both in terms of signal quality and usability. In particular, the three types consisted in gold-coated single pin, multiple pins and solid-gel electrodes. The results confirmed the great standards achieved by dry electrode industry, since it was possible to obtain results comparable to wet electrodes in terms of signals spectra and mental states classification, but at the same time drastically reducing the time of montage and enhancing the comfort. In particular, multiple-pins and solid-gel electrodes overcome gold-coated single-pin-based ones in terms of comfort.

Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects

Recent advances in neuroscience have paved the way to innovative applications that cognitively augment and enhance humans in a variety of contexts. This paper aims at providing a snapshot of the current state of the art and a motivated forecast of the most likely developments in the next two decades. Firstly, we survey the main neuroscience technologies for both observing and influencing brain activity, which are necessary ingredients for human cognitive augmentation. We also compare and contrast such technologies, as their individual characteristics (e.g., spatio-temporal resolution, invasiveness, portability, energy requirements, and cost) influence their current and future role in human cognitive augmentation. Secondly, we chart the state of the art on neurotechnologies for human cognitive augmentation, keeping an eye both on the applications that already exist and those that are emerging or are likely to emerge in the next two decades. Particularly, we consider applications in the areas of communication, cognitive enhancement, memory, attention monitoring/enhancement, situation awareness and complex problem solving, and we look at what fraction of the population might benefit from such technologies and at the demands they impose in terms of user training. Thirdly, we briefly review the ethical issues associated with current neuroscience technologies. These are important because they may differentially influence both present and future research on (and adoption of) neurotechnologies for human cognitive augmentation: an inferior technology with no significant ethical issues may thrive while a superior technology causing widespread ethical concerns may end up being outlawed. Finally, based on the lessons learned in our analysis, using past trends and considering other related forecasts, we attempt to forecast the most likely future developments of neuroscience technology for human cognitive augmentation and provide informed recommendations for promising future research and exploitation avenues.

EEG-Based Mental Workload Neurometric to Evaluate the Impact of Different Traffic and Road Conditions in Real Driving Settings

Car driving is considered a very complex activity, consisting of different concomitant tasks and subtasks, thus it is crucial to understand the impact of different factors, such as road complexity, traffic, dashboard devices, and external events on the driver's behavior and performance. For this reason, in particular situations the cognitive demand experienced by the driver could be very high, inducing an excessive experienced mental workload and consequently an increasing of error commission probability. In this regard, it has been demonstrated that human error is the main cause of the 57% of road accidents and a contributing factor in most of them. In this study, 20 young subjects have been involved in a real driving experiment, performed under different traffic conditions (rush hour and not) and along different road types (main and secondary streets). Moreover, during the driving tasks different specific events, in particular a pedestrian crossing the road and a car entering the traffic flow just ahead of the experimental subject, have been acted. A Workload Index based on the Electroencephalographic (EEG), i.e., brain activity, of the drivers has been employed to investigate the impact of the different factors on the driver's workload. Eye-Tracking (ET) technology and subjective measures have also been employed in order to have a comprehensive overview of the driver's perceived workload and to investigate the different insights obtainable from the employed methodologies. The employment of such EEG-based Workload index confirmed the significant impact of both traffic and road types on the drivers' behavior (increasing their workload), with the advantage of being under real settings. Also, it allowed to highlight the increased workload related to external events while driving, in particular with a significant effect during those situations when the traffic was low. Finally, the comparison between methodologies revealed the higher sensitivity of neurophysiological measures with respect to ET and subjective ones. In conclusion, such an EEG-based Workload index would allow to assess objectively the mental workload experienced by the driver, standing out as a powerful tool for research aimed to investigate drivers' behavior and providing additional and complementary insights with respect to traditional methodologies employed within road safety research.

Investigating Neural Sensorimotor Mechanisms Underlying Flight Expertise in Pilots: Preliminary Data From an EEG Study

Over the last decade, the efforts toward unraveling the complex interplay between the brain, body, and environment have set a promising line of research that utilizes neuroscience to study human performance in natural work contexts such as aviation. Thus, a relatively new discipline called neuroergonomics is holding the promise of studying the neural mechanisms underlying human performance in pursuit of both theoretical and practical insights. In this work, we utilized a neuroergonomic approach by combining insights from ecological psychology and embodied cognition to study flight expertise. Specifically, we focused on the Mirror Neuron system as a key correlate for understanding the interaction between an individual and the environment, suggesting that it can be used to index changes in the coupling of perception-action associated with skill development. In this study, we measured the EEG mu suppression as a proxy of the Mirror Neuron system in experts (pilots) and novices while performing a distance estimation task in a landing scenario. To survey the specificity of this measure, we considered central, parietal and occipital electrode pools and analyzed alpha (8–13 Hz) and beta (18–25 Hz) rhythm bands. We hypothesized that in experts vs. novices, specific neural sensorimotor brain activity would underpin the connection between perception and action in an in-flight context. Preliminary results indicate that alpha and beta rhythm suppression was area-specific irrespective of groups, present in the central electrodes placed over the motor areas. Group analysis revealed that specifically alpha mu rhythm, but not beta, was significantly more suppressed in pilots vs. novices. Complementing these findings we found a trend in which the strength of mu suppression increased with the sense of presence experienced by the pilots. Such sensorimotor activation is in line with the idea that for a pilot, a distance judgment is intimately associated with the function of landing. This reflects the ability to use optical invariants to see the world in terms of the capabilities of the aircraft (e.g., reachability and glide angle). These preliminary findings support the role of embodied simulation mechanisms in visual perception and add important insights into a practical understanding of flight expertise, suggesting sensorimotor mechanisms as potential neuro-markers.

Neurophysiological Responses to Different Product Experiences

It is well known that the evaluation of a product from the shelf considers the simultaneous cerebral and emotional evaluation of the different qualities of the product such as its colour, the eventual images shown, and the envelope's texture (hereafter all included in the term “product experience”). However, the measurement of cerebral and emotional reactions during the interaction with food products has not been investigated in depth in specialized literature. The aim of this paper was to investigate such reactions by the EEG and the autonomic activities, as elicited by the cross-sensory interaction (sight and touch) across several different products. In addition, we investigated whether (i) the brand (Major Brand or Private Label), (ii) the familiarity (Foreign or Local Brand), and (iii) the hedonic value of products (Comfort Food or Daily Food) influenced the reaction of a group of volunteers during their interaction with the products. Results showed statistically significantly higher tendency of cerebral approach (as indexed by EEG frontal alpha asymmetry) in response to comfort food during the visual exploration and the visual and tactile exploration phases. Furthermore, for the same index, a higher tendency of approach has been found toward foreign food products in comparison with local food products during the visual and tactile exploration phase. Finally, the same comparison performed on a different index (EEG frontal theta) showed higher mental effort during the interaction with foreign products during the visual exploration and the visual and tactile exploration phases. Results from the present study could deepen the knowledge on the neurophysiological response to food products characterized by different nature in terms of hedonic value familiarity; moreover, they could have implications for food marketers and finally lead to further study on how people make food choices through the interactions with their commercial envelope.

How the workload impacts on cognitive cooperation: A pilot study

Cooperation degradation can be seen as one of the main causes of human errors. Poor cooperation could arise from aberrant mental processes, such as mental overload, that negatively affect the user's performance. Using different levels of difficulty in a cooperative task, we combined behavioural, subjective and neurophysiological data with the aim to i) quantify the mental workload under which the crew was operating, ii) evaluate the degree of their cooperation, and iii) assess the impact of the workload demands on the cooperation levels. The combination of such data showed that high workload demand impacted significantly on the performance, workload perception, and degree of cooperation.

Brain Interaction during Cooperation: Evaluating Local Properties of Multiple-Brain Network

Subjects’ interaction is the core of most human activities. This is the reason why a lack of coordination is often the cause of missing goals, more than individual failure. While there are different subjective and objective measures to assess the level of mental effort required by subjects while facing a situation that is getting harder, that is, mental workload, to define an objective measure based on how and if team members are interacting is not so straightforward. In this study, behavioral, subjective and synchronized electroencephalographic data were collected from couples involved in a cooperative task to describe the relationship between task difficulty and team coordination, in the sense of interaction aimed at cooperatively performing the assignment. Multiple-brain connectivity analysis provided information about the whole interacting system. The results showed that averaged local properties of a brain network were affected by task difficulty. In particular, strength changed significantly with task difficulty and clustering coefficients strongly correlated with the workload itself. In particular, a higher workload corresponded to lower clustering values over the central and parietal brain areas. Such results has been interpreted as less efficient organization of the network when the subjects’ activities, due to high workload tendencies, were less coordinated.