Sensor Networks for Aerospace Human-Machine Systems

Integrating physiological monitoring systems in military aviation: a brief narrative review of its importance, opportunities, and risks

Military pilots risk their lives during training and operations. Advancements in aerospace engineering, flight profiles, and mission demands may require the pilot to test the safe limits of their physiology. Monitoring pilot physiology (e.g. heart rate, oximetry, and respiration) inflight is in consideration by several nations to inform pilots of reduced performance capacity and guide future developments in aircraft and life-support system design. Numerous challenges, however, prevent the immediate operationalisation of physiological monitoring sensors, particularly their unreliability in the aerospace environment and incompatibility with pilot clothing and protective equipment. Human performance and behaviour are also highly variable and measuring these in controlled laboratory settings do not mirror the real-world conditions pilots must endure. Misleading or erroneous predictive models are unacceptable as these could compromise mission success and lose operator trust. This narrative review provides an overview of considerations for integrating physiological monitoring systems within the military aviation environment.Practitioner summary: Advancements in military technology can conflictingly enhance and compromise pilot safety and performance. We summarise some of the opportunities, limitations, and risks of integrating physiological monitoring systems within military aviation. Our intent is to catalyse further research and technological development.Abbreviations: AGS: anti-gravity suit; AGSM: anti-gravity straining manoeuvre; A-LOC: almost loss of consciousness; CBF: cerebral blood flow; ECG: electrocardiogram; EEG: electroencephalogram; fNIRS: functional near-infrared spectroscopy; G-forces: gravitational forces; G-LOC: gravity-induced loss of consciousness; HR: heart rate; HRV: heart rate variability; LSS: life-support system; NATO: North Atlantic Treaty Organisation; PE: Physiological Episode; PCO2: partial pressure of carbon dioxide; PO2: partial pressure of oxygen; OBOGS: on board oxygen generating systems; SpO2: peripheral blood haemoglobin-oxygen saturation; STANAG: North Atlantic Treaty Organisation Standardisation Agreement; UPE: Unexplained Physiological Episode; WBV: whole body vibration.

Stress and Workload Assessment in Aviation-A Narrative Review

In aviation, any detail can have massive consequences. Among the potential sources of failure, human error is still the most troublesome to handle. Therefore, research concerning the management of mental workload, attention, and stress is of special interest in aviation. Recognizing conditions in which a pilot is over-challenged or cannot act lucidly could avoid serious outcomes. Furthermore, knowing in depth a pilot's neurophysiological and cognitive-behavioral responses could allow for the optimization of equipment and procedures to minimize risk and increase safety. In addition, it could translate into a general enhancement of both the physical and mental well-being of pilots, producing a healthier and more ergonomic work environment. This review brings together literature on the study of stress and workload in the specific case of pilots of both civil and military aircraft. The most common approaches for studying these phenomena in the avionic context are explored in this review, with a focus on objective methodologies (e.g., the collection and analysis of neurophysiological signals). This review aims to identify the pros, cons, and applicability of the various approaches, to enable the design of an optimal protocol for a comprehensive study of these issues.

Wearable Cardiorespiratory Sensors for Aerospace Applications

Emerging Air Traffic Management (ATM) and avionics human-machine system concepts require the real-time monitoring of the human operator to support novel task assessment and system adaptation features. To realise these advanced concepts, it is essential to resort to a suite of sensors recording neurophysiological data reliably and accurately. This article presents the experimental verification and performance characterisation of a cardiorespiratory sensor for ATM and avionics applications. In particular, the processed physiological measurements from the designated commercial device are verified against clinical-grade equipment. Compared to other studies which only addressed physical workload, this characterisation was performed also looking at cognitive workload, which poses certain additional challenges to cardiorespiratory monitors. The article also addresses the quantification of uncertainty in the cognitive state estimation process as a function of the uncertainty in the input cardiorespiratory measurements. The results of the sensor verification and of the uncertainty propagation corroborate the basic suitability of the commercial cardiorespiratory sensor for the intended aerospace application but highlight the relatively poor performance in respiratory measurements during a purely mental activity.

Explanation of Machine-Learning Solutions in Air-Traffic Management

Advances in the trusted autonomy of air-traffic management (ATM) systems are currently being pursued to cope with the predicted growth in air-traffic densities in all classes of airspace. Highly automated ATM systems relying on artificial intelligence (AI) algorithms for anomaly detection, pattern identification, accurate inference, and optimal conflict resolution are technically feasible and demonstrably able to take on a wide variety of tasks currently accomplished by humans. However, the opaqueness and inexplicability of most intelligent algorithms restrict the usability of such technology. Consequently, AI-based ATM decision-support systems (DSS) are foreseen to integrate eXplainable AI (XAI) in order to increase interpretability and transparency of the system reasoning and, consequently, build the human operators’ trust in these systems. This research presents a viable solution to implement XAI in ATM DSS, providing explanations that can be appraised and analysed by the human air-traffic control operator (ATCO). The maturity of XAI approaches and their application in ATM operational risk prediction is investigated in this paper, which can support both existing ATM advisory services in uncontrolled airspace (Classes E and F) and also drive the inflation of avoidance volumes in emerging performance-driven autonomy concepts. In particular, aviation occurrences and meteorological databases are exploited to train a machine learning (ML)-based risk-prediction tool capable of real-time situation analysis and operational risk monitoring. The proposed approach is based on the XGBoost library, which is a gradient-boost decision tree algorithm for which post-hoc explanations are produced by SHapley Additive exPlanations (SHAP) and Local Interpretable Model-Agnostic Explanations (LIME). Results are presented and discussed, and considerations are made on the most promising strategies for evolving the human–machine interactions (HMI) to strengthen the mutual trust between ATCO and systems. The presented approach is not limited only to conventional applications but also suitable for UAS-traffic management (UTM) and other emerging applications.

Online Multimodal Inference of Mental Workload for Cognitive Human Machine Systems

With increasingly higher levels of automation in aerospace decision support systems, it is imperative that the human operator maintains a high level of situational awareness in different operational conditions and a central role in the decision-making process. While current aerospace systems and interfaces are limited in their adaptability, a Cognitive Human Machine System (CHMS) aims to perform dynamic, real-time system adaptation by estimating the cognitive states of the human operator. Nevertheless, to reliably drive system adaptation of current and emerging aerospace systems, there is a need to accurately and repeatably estimate cognitive states, particularly for Mental Workload (MWL), in real-time. As part of this study, two sessions were performed during a Multi-Attribute Task Battery (MATB) scenario, including a session for offline calibration and validation and a session for online validation of eleven multimodal inference models of MWL. The multimodal inference model implemented included an Adaptive Neuro Fuzzy Inference System (ANFIS), which was used in different configurations to fuse data from an Electroencephalogram (EEG) model’s output, four eye activity features and a control input feature. The results from the online validation of the ANFIS models demonstrated that five of the ANFIS models (containing different feature combinations of eye activity and control input features) all demonstrated good results, while the best performing model (containing all four eye activity features and the control input feature) showed an average Mean Absolute Error (MAE) = 0.67 ± 0.18 and Correlation Coefficient (CC) = 0.71 ± 0.15. The remaining six ANFIS models included data from the EEG model’s output, which had an offset discrepancy. This resulted in an equivalent offset for the online multimodal fusion. Nonetheless, the efficacy of these ANFIS models could be seen with the pairwise correlation with the task level, where one model demonstrated a CC = 0.77 ± 0.06, which was the highest among all the ANFIS models tested. Hence, this study demonstrates the ability for online multimodal fusion from features extracted from EEG signals, eye activity and control inputs to produce an accurate and repeatable inference of MWL.

Adaptive Human-Robot Interactions for Multiple Unmanned Aerial Vehicles

Advances in unmanned aircraft systems (UAS) have paved the way for progressively higher levels of intelligence and autonomy, supporting new modes of operation, such as the one-to-many (OTM) concept, where a single human operator is responsible for monitoring and coordinating the tasks of multiple unmanned aerial vehicles (UAVs). This paper presents the development and evaluation of cognitive human-machine interfaces and interactions (CHMI2) supporting adaptive automation in OTM applications. A CHMI2 system comprises a network of neurophysiological sensors and machine-learning based models for inferring user cognitive states, as well as the adaptation engine containing a set of transition logics for control/display functions and discrete autonomy levels. Models of the user’s cognitive states are trained on past performance and neurophysiological data during an offline calibration phase, and subsequently used in the online adaptation phase for real-time inference of these cognitive states. To investigate adaptive automation in OTM applications, a scenario involving bushfire detection was developed where a single human operator is responsible for tasking multiple UAV platforms to search for and localize bushfires over a wide area. We present the architecture and design of the UAS simulation environment that was developed, together with various human-machine interface (HMI) formats and functions, to evaluate the CHMI2 system’s feasibility through human-in-the-loop (HITL) experiments. The CHMI2 module was subsequently integrated into the simulation environment, providing the sensing, inference, and adaptation capabilities needed to realise adaptive automation. HITL experiments were performed to verify the CHMI2 module’s functionalities in the offline calibration and online adaptation phases. In particular, results from the online adaptation phase showed that the system was able to support real-time inference and human-machine interface and interaction (HMI2) adaptation. However, the accuracy of the inferred workload was variable across the different participants (with a root mean squared error (RMSE) ranging from 0.2 to 0.6), partly due to the reduced number of neurophysiological features available as real-time inputs and also due to limited training stages in the offline calibration phase. To improve the performance of the system, future work will investigate the use of alternative machine learning techniques, additional neurophysiological input features, and a more extensive training stage.

How Can Physiological Computing Benefit Human-Robot Interaction?

As systems grow more automatized, the human operator is all too often overlooked. Although human-robot interaction (HRI) can be quite demanding in terms of cognitive resources, the mental states (MS) of the operators are not yet taken into account by existing systems. As humans are no providential agents, this lack can lead to hazardous situations. The growing number of neurophysiology and machine learning tools now allows for efficient operators’ MS monitoring. Sending feedback on MS in a closed-loop solution is therefore at hand. Involving a consistent automated planning technique to handle such a process could be a significant asset. This perspective article was meant to provide the reader with a synthesis of the significant literature with a view to implementing systems that adapt to the operator’s MS to improve human-robot operations’ safety and performance. First of all, the need for this approach is detailed regarding remote operation, an example of HRI. Then, several MS identified as crucial for this type of HRI are defined, along with relevant electrophysiological markers. A focus is made on prime degraded MS linked to time-on-task and task demands, as well as collateral MS linked to system outputs (i.e., feedback and alarms). Lastly, the principle of symbiotic HRI is detailed and one solution is proposed to include the operator state vector into the system using a mixed-initiative decisional framework to drive such an interaction.

A Cyber-Physical-Human System for One-to-Many UAS Operations: Cognitive Load Analysis

The continuing development of avionics for Unmanned Aircraft Systems (UASs) is introducing higher levels of intelligence and autonomy both in the flight vehicle and in the ground mission control, allowing new promising operational concepts to emerge. One-to-Many (OTM) UAS operations is one such concept and its implementation will require significant advances in several areas, particularly in the field of Human-Machine Interfaces and Interactions (HMI2). Measuring cognitive load during OTM operations, in particular Mental Workload (MWL), is desirable as it can relieve some of the negative effects of increased automation by providing the ability to dynamically optimize avionics HMI2 to achieve an optimal sharing of tasks between the autonomous flight vehicles and the human operator. The novel Cognitive Human Machine System (CHMS) proposed in this paper is a Cyber-Physical Human (CPH) system that exploits the recent technological developments of affordable physiological sensors. This system focuses on physiological sensing and Artificial Intelligence (AI) techniques that can support a dynamic adaptation of the HMI2 in response to the operators' cognitive state (including MWL), external/environmental conditions and mission success criteria. However, significant research gaps still exist, one of which relates to a universally valid method for determining MWL that can be applied to UAS operational scenarios. As such, in this paper we present results from a study on measuring MWL on five participants in an OTM UAS wildfire detection scenario, using Electroencephalogram (EEG) and eye tracking measurements. These physiological data are compared with a subjective measure and a task index collected from mission-specific data, which serves as an objective task performance measure. The results show statistically significant differences for all measures including the subjective, performance and physiological measures performed on the various mission phases. Additionally, a good correlation is found between the two physiological measurements and the task index. Fusing the physiological data and correlating with the task index gave the highest correlation coefficient (CC = 0.726 ± 0.14) across all participants. This demonstrates how fusing different physiological measurements can provide a more accurate representation of the operators' MWL, whilst also allowing for increased integrity and reliability of the system.

Aircraft Pilots Workload Analysis: Heart Rate Variability Objective Measures and NASA-Task Load Index Subjective Evaluation

Workload and fatigue of aircraft pilots represent an argument of great interest in the framework of human factors and a pivotal point to be considered in aviation safety. 75% of aircraft accidents are related to human errors that, in most cases, are due to high level of mental workload and fatigue. There exist several subjective or objective metrics to quantify the pilots’ workload level, with both linear and nonlinear relationships reported in the literature. The main research objective of the present work is to analyze the relationships between objective and subjective workload measurements by looking for a correlation between metrics belonging to the subjective and biometric rating methods. More particularly, the Heart Rate Variability (HRV) is used for the objective analysis, whereas the NASA-TLX questionnaire is the tool chosen for the subjective evaluation of the workload. Two different flight scenarios were considered for the studies: the take-off phase with the initial climb and the final approach phase with the landing. A Maneuver Error Index (MEI) is also introduced to evaluate the pilot flight performance according to mission requirements. Both qualitative and quantitative correlation analyses were performed among the MEI, subjective and objective measurements. Monotonic relationships were found within the HRV indexes, and a nonlinear relationship is proposed among NASA-TLX and HRV indexes. These findings suggest that the relationship between workload, biometric data, and performance indexes are characterized by intricate patterns of nonlinear relationships.

Monitoring the Risk of the Electric Component Imposed on a Pilot During Light Aircraft Operations in a High-Frequency Electromagnetic Field

High-frequency electromagnetic fields can have a negative effect on both the human body and electronic devices. The devices and systems utilized in radio communications constitute the most numerous sources of electromagnetic fields. The following research investigates values of the electric component of electromagnetic field intensification determined with the ESM 140 dosimeter during the flights of four aircrafts—Cessna C152, Cessna C172, Aero AT3 R100, and Robinson R44 Raven helicopter—from the airport in Depultycze Krolewskie near Chelm, Poland. The point of reference for the obtained results were the normative limits of the electromagnetic field that can affect a pilot in the course of a flight. The maximum value registered by the dosimeter was E = 3.307 V/m for GSM 1800 frequencies.

Uncertainty Quantification for Space Situational Awareness and Traffic Management

This paper presents a sensor-orientated approach to on-orbit position uncertainty generation and quantification for both ground-based and space-based surveillance applications. A mathematical framework based on the least squares formulation is developed to exploit real-time navigation measurements and tracking observables to provide a sound methodology that supports separation assurance and collision avoidance among Resident Space Objects (RSO). In line with the envisioned Space Situational Awareness (SSA) evolutions, the method aims to represent the navigation and tracking errors in the form of an uncertainty volume that accurately depicts the size, shape, and orientation. Simulation case studies are then conducted to verify under which sensors performance the method meets Gaussian assumptions, with a greater view to the implications that uncertainty has on the cyber-physical architecture evolutions and Cognitive Human-Machine Systems required for Space Situational Awareness and the development of a comprehensive Space Traffic Management framework.