1
|
Festa EK, Bracken BK, Desrochers PC, Winder AT, Strong PK, Endsley MR. EEG and fNIRS are associated with situation awareness (hazard) prediction during a driving task. ERGONOMICS 2024:1-16. [PMID: 38899938 DOI: 10.1080/00140139.2024.2367163] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 11/23/2023] [Accepted: 06/06/2024] [Indexed: 06/21/2024]
Abstract
Situation awareness (SA) is important in many demanding tasks (e.g. driving). Assessing SA during training can indicate whether someone is ready to perform in the real world. SA is typically assessed by interrupting the task to ask questions about the situation or asking questions after task completion, assessing only momentary SA. An objective and continuous means of detecting SA is needed. We examined whether neurophysiological sensors are useful to objectively measure Level 3 SA (projection of events into the future) during a driving task. We measured SA by the speed at which participants responded to SA questions and the accuracy of responses. For EEG, beta and theta power were most sensitive to SA response time. For fNIRS, oxygenated haemoglobin (HbO) was most sensitive to accuracy. This is the first evidence to our knowledge that neurophysiological measures are useful for assessing Level 3 SA during an ecologically valid task.
Collapse
Affiliation(s)
- Elena K Festa
- Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI, USA
| | | | | | | | - Peyton K Strong
- Cognitive, Linguistic, and Psychological Sciences, Brown University, Providence, RI, USA
| | | |
Collapse
|
2
|
Enders LR, Gordon SM, Roy H, Rohaly T, Dalangin B, Jeter A, Villarreal J, Boykin GL, Touryan J. Evidence of elevated situational awareness for active duty soldiers during navigation of a virtual environment. PLoS One 2024; 19:e0298867. [PMID: 38728266 PMCID: PMC11086823 DOI: 10.1371/journal.pone.0298867] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/12/2023] [Accepted: 02/01/2024] [Indexed: 05/12/2024] Open
Abstract
U.S. service members maintain constant situational awareness (SA) due to training and experience operating in dynamic and complex environments. Work examining how military experience impacts SA during visual search of a complex naturalistic environment, is limited. Here, we compare Active Duty service members and Civilians' physiological behavior during a navigational visual search task in an open-world virtual environment (VE) while cognitive load was manipulated. We measured eye-tracking and electroencephalogram (EEG) outcomes from Active Duty (N = 21) and Civilians (N = 15) while they navigated a desktop VE at a self-regulated pace. Participants searched and counted targets (N = 15) presented among distractors, while cognitive load was manipulated with an auditory Math Task. Results showed Active Duty participants reported significantly greater/closer to the correct number of targets compared to Civilians. Overall, Active Duty participants scanned the VE with faster peak saccade velocities and greater average saccade magnitudes compared to Civilians. Convolutional Neural Network (CNN) response (EEG P-300) was significantly weighted more to initial fixations for the Active Duty group, showing reduced attentional resources on object refixations compared to Civilians. There were no group differences in fixation outcomes or overall CNN response when comparing targets versus distractor objects. When cognitive load was manipulated, only Civilians significantly decreased their average dwell time on each object and the Active Duty group had significantly fewer numbers of correct answers on the Math Task. Overall, the Active Duty group explored the VE with increased scanning speed and distance and reduced cognitive re-processing on objects, employing a different, perhaps expert, visual search strategy indicative of increased SA. The Active Duty group maintained SA in the main visual search task and did not appear to shift focus to the secondary Math Task. Future work could compare how a stress inducing environment impacts these groups' physiological or cognitive markers and performance for these groups.
Collapse
Affiliation(s)
- Leah R. Enders
- Human in Complex Systems Division, DEVCOM Army Research Laboratory, Aberdeen Proving Ground, Maryland, United States of America
| | | | - Heather Roy
- Human in Complex Systems Division, DEVCOM Army Research Laboratory, Aberdeen Proving Ground, Maryland, United States of America
| | - Thomas Rohaly
- DCS Corporation, Alexandria, Virginia, United States of America
| | - Bianca Dalangin
- DCS Corporation, Alexandria, Virginia, United States of America
| | - Angela Jeter
- DCS Corporation, Alexandria, Virginia, United States of America
| | | | - Gary L. Boykin
- Human in Complex Systems Division, DEVCOM Army Research Laboratory, Aberdeen Proving Ground, Maryland, United States of America
| | - Jonathan Touryan
- Human in Complex Systems Division, DEVCOM Army Research Laboratory, Aberdeen Proving Ground, Maryland, United States of America
| |
Collapse
|
3
|
Chen J, Chen A, Jiang B, Zhang X. Physiological records-based situation awareness evaluation under aviation context: A comparative analysis. Heliyon 2024; 10:e26409. [PMID: 38434275 PMCID: PMC10907521 DOI: 10.1016/j.heliyon.2024.e26409] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 01/23/2024] [Accepted: 02/13/2024] [Indexed: 03/05/2024] Open
Abstract
Situational Awareness (SA) assessment is of paramount importance in various domains, with particular significance in the military for safe aviation decision-making. It involves encompassing perception, comprehension, and projection levels in human beings. Accurate evaluation of SA statuses across these three levels is crucial for mitigating human false-positive and false-negative rates in monitoring complex scenarios in the aviation context. This study proposes a comprehensive comparative analysis by involving two types of physiological records: electroencephalogram (EEG) signals and brain electrical activity mapping (BEAM) images. These two modalities are leveraged to automate precise SA evaluation using both conventional machine learning and advanced deep learning techniques. Benchmarking experiments reveal that the BEAM-based deep learning models attain state-of-the-art performance scores of 0.955 for both SA perception and comprehension levels, respectively. Conversely, the EEG signals-based manual feature extraction, selection, and classification approach achieved a superior accuracy of 0.929 for the projection level of SA. These findings collectively highlight the potential of deploying diverse physiological records as valuable computational tools for enhancing SA evaluation throughout aviation decision-making safety.
Collapse
Affiliation(s)
- Jun Chen
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, 710072, China
- Chongqing Institute for Brain and Intelligence, Guangyang Bay Laboratory, Chongqing, 400064, China
| | - Anqi Chen
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, 710072, China
| | - Bingkun Jiang
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, 710072, China
| | - Xinyu Zhang
- School of Electronics and Information, Northwestern Polytechnical University, Xi'an, 710072, China
| |
Collapse
|
4
|
Li X, Kang Y, Chen W, Liu F, Jiao Y, Luo Y. Recognizing the situation awareness of forklift operators based on EEG techniques in a field experiment. Front Neurosci 2024; 18:1323190. [PMID: 38445257 PMCID: PMC10912158 DOI: 10.3389/fnins.2024.1323190] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/17/2023] [Accepted: 01/30/2024] [Indexed: 03/07/2024] Open
Abstract
Lack of situation awareness (SA) is the primary cause of human errors when operating forklifts, so determining the SA level of the forklift operator is crucial to the safety of forklift operations. An EEG recognition approach of forklift operator SA in actual settings was presented in order to address the issues with invasiveness, subjectivity, and intermittency of existing measuring methods. In this paper, we conducted a field experiment that mimicked a typical forklift operation scenario to verify the differences in EEG states of forklift operators with different SA levels and investigate the correlation of multi-band combination features of each brain region of forklift operators with SA. Based on the sensitive EEG combination indexes, Support Vector Mechanism was used to construct a forklift operator SA recognition model. The results revealed that there were differences between forklift operators with high and low SA in the θ, α, and β frequency bands in zones F, C, P, and O; combined EEG indicators θ/β, (α + θ)/(α + β), and θ/(α + β) in zones F, P, and C were significantly correlated with SA; the recognition accuracy of the model reached 88.64% in the case of combined EEG indicators of zones C & F & P as input. It could provide a reference for SA measurement, contributing to the improvement of SA.
Collapse
Affiliation(s)
- Xin Li
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
- COSCO SHIPPING Heavy Industry Co., Ltd., Shanghai, China
| | - Yutao Kang
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
| | - Weijiong Chen
- Merchant Marine College, Shanghai Maritime University, Shanghai, China
| | - Feng Liu
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
| | - Yu Jiao
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
| | - Yabin Luo
- College of Ocean Science and Engineering, Shanghai Maritime University, Shanghai, China
| |
Collapse
|
5
|
Liu C, Zhang C, Sun L, Liu K, Liu H, Zhu W, Jiang C. Detection of Pilot's Mental Workload Using a Wireless EEG Headset in Airfield Traffic Pattern Tasks. ENTROPY (BASEL, SWITZERLAND) 2023; 25:1035. [PMID: 37509982 PMCID: PMC10378707 DOI: 10.3390/e25071035] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Subscribe] [Scholar Register] [Received: 06/07/2023] [Revised: 06/25/2023] [Accepted: 07/05/2023] [Indexed: 07/30/2023]
Abstract
Elevated mental workload (MWL) experienced by pilots can result in increased reaction times or incorrect actions, potentially compromising flight safety. This study aims to develop a functional system to assist administrators in identifying and detecting pilots' real-time MWL and evaluate its effectiveness using designed airfield traffic pattern tasks within a realistic flight simulator. The perceived MWL in various situations was assessed and labeled using NASA Task Load Index (NASA-TLX) scores. Physiological features were then extracted using a fast Fourier transformation with 2-s sliding time windows. Feature selection was conducted by comparing the results of the Kruskal-Wallis (K-W) test and Sequential Forward Floating Selection (SFFS). The results proved that the optimal input was all PSD features. Moreover, the study analyzed the effects of electroencephalography (EEG) features from distinct brain regions and PSD changes across different MWL levels to further assess the proposed system's performance. A 10-fold cross-validation was performed on six classifiers, and the optimal accuracy of 87.57% was attained using a multi-class K-Nearest Neighbor (KNN) classifier for classifying different MWL levels. The findings indicate that the wireless headset-based system is reliable and feasible. Consequently, numerous wireless EEG device-based systems can be developed for application in diverse real-driving scenarios. Additionally, the current system contributes to future research on actual flight conditions.
Collapse
Affiliation(s)
- Chenglin Liu
- School of Transportation & Logistics, Southwest Jiaotong University, Chengdu 611756, China
| | - Chenyang Zhang
- School of Transportation & Logistics, Southwest Jiaotong University, Chengdu 611756, China
| | - Luohao Sun
- School of Information Science and Technology, Southwest Jiaotong University, Chengdu 611756, China
| | - Kun Liu
- School of Transportation & Logistics, Southwest Jiaotong University, Chengdu 611756, China
| | - Haiyue Liu
- School of Transportation & Logistics, Southwest Jiaotong University, Chengdu 611756, China
| | - Wenbing Zhu
- School of Transportation & Logistics, Southwest Jiaotong University, Chengdu 611756, China
| | - Chaozhe Jiang
- School of Transportation & Logistics, Southwest Jiaotong University, Chengdu 611756, China
| |
Collapse
|
6
|
Jiang S, Chen W, Ren Z, Zhu H. EEG-based analysis for pilots' at-risk cognitive competency identification using RF-CNN algorithm. Front Neurosci 2023; 17:1172103. [PMID: 37152589 PMCID: PMC10160375 DOI: 10.3389/fnins.2023.1172103] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/23/2023] [Accepted: 04/03/2023] [Indexed: 05/09/2023] Open
Abstract
Cognitive competency is an essential complement to the existing ship pilot screening system that should be focused on. Situation awareness (SA), as the cognitive foundation of unsafe behaviors, is susceptible to influencing piloting performance. To address this issue, this paper develops an identification model based on random forest- convolutional neural network (RF-CNN) method for detecting at-risk cognitive competency (i.e., low SA level) using wearable EEG signal acquisition technology. In the poor visibility scene, the pilots' SA levels were correlated with EEG frequency metrics in frontal (F) and central (C) regions, including α/β (p = 0.071 < 0.1 in F and p = 0.042 < 0.05 in C), θ/(α + θ) (p = 0.048 < 0.05 in F and p = 0.026 < 0.05 in C) and (α + θ)/β (p = 0.046 < 0.05 in F and p = 0.012 < 0.05 in C), and then a total of 12 correlation features were obtained based on a 5 s sliding time window. Using the RF algorithm developed by principal component analysis (PCA) for further feature combination, these salient combinations are used as input sets to obtain the CNN algorithm with optimal parameters for identification. The comparative results of the proposed RF-CNN (accuracy is 84.8%) against individual RF (accuracy is 78.1%) and CNN (accuracy is 81.6%) methods demonstrate that the RF-CNN with feature optimization provides the best identification of at-risk cognitive competency (accuracy increases 6.7%). Overall, the results of this paper provide key technical support for the development of an adaptive evaluation system of pilots' cognitive competency based on intelligent technology, and lay the foundation and framework for monitoring the cognitive process and competency of ship piloting operation in China.
Collapse
Affiliation(s)
- Shaoqi Jiang
- College of Information Engineering, Jinhua Polytechnic, Jinhua, Zhejiang, China
- College of Environment and Engineering, Shanghai Maritime University, Shanghai, China
- *Correspondence: Shaoqi Jiang,
| | - Weijiong Chen
- College of Environment and Engineering, Shanghai Maritime University, Shanghai, China
- College of Merchant Marine, Shanghai Maritime University, Shanghai, China
| | - Zhenzhen Ren
- College of Merchant Marine, Shanghai Maritime University, Shanghai, China
| | - He Zhu
- College of Information Engineering, Jinhua Polytechnic, Jinhua, Zhejiang, China
| |
Collapse
|
7
|
Recognising situation awareness associated with different workloads using EEG and eye-tracking features in air traffic control tasks. Knowl Based Syst 2022. [DOI: 10.1016/j.knosys.2022.110179] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/12/2022]
|
8
|
Lu L, Xie Z, Wang H, Li L, Xu X. Mental stress and safety awareness during human-robot collaboration - Review. APPLIED ERGONOMICS 2022; 105:103832. [PMID: 35772289 DOI: 10.1016/j.apergo.2022.103832] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.5] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 07/06/2021] [Revised: 06/14/2022] [Accepted: 06/15/2022] [Indexed: 06/15/2023]
Abstract
Human-robot collaboration (HRC) is an emerging research area that has gained tremendous attention in both academia and industry. Yet, the feature that humans and robots sharing the workplace has led to safety concerns. In particular, the mental stress or safety awareness of human teammates during HRC remains unclear but is also of great importance to workplace safety. In this manuscript, we reviewed twenty-five studies for understanding the relationships between HRC and workers' mental stress or safety awareness. Specifically, we aimed to understand: (1) robot-related factors that may affect human workers' mental stress or safety awareness, (2) a number of measurements that could be used to evaluate workers' mental stress in HRC, and (3) various methods for measuring safety awareness that had been adopted or could be applied in HRC. According to our literature review, robot-related factors including robot characteristics, social touching and trajectory have relationships with workers' mental stress or safety awareness. For the measurement of mental stress and safety awareness, each method mentioned has its validity and rationality. Additionally, a discussion related to the potential co-robot actions to lower mental stress or improve safety awareness as well as future implications were provided.
Collapse
Affiliation(s)
- Lu Lu
- Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh NC, 27695, USA
| | - Ziyang Xie
- Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh NC, 27695, USA
| | - Hanwen Wang
- Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh NC, 27695, USA
| | - Li Li
- Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh NC, 27695, USA
| | - Xu Xu
- Edward P. Fitts Department of Industrial and Systems Engineering, North Carolina State University, Raleigh NC, 27695, USA.
| |
Collapse
|
9
|
Wan Z, Yang R, Huang M, Liu W, Zeng N. EEG fading data classification based on improved manifold learning with adaptive neighborhood selection. Neurocomputing 2022. [DOI: 10.1016/j.neucom.2021.11.039] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/19/2022]
|
10
|
A Digital Twin-Based Platform towards Intelligent Automation with Virtual Counterparts of Flight and Air Traffic Control Operations. APPLIED SCIENCES-BASEL 2021. [DOI: 10.3390/app112210923] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/16/2022]
Abstract
Automation technologies have been deployed widely to boost the efficiency of production and operations, to trim the complicated process, and to reduce the human error involved. Nevertheless, aviation remains human-centred and requires collaboration between different parties. Given the lack of a collaborative decision-making training platform for air traffic operations in the industry, this study utilises the concept of cyber-physical systems (CPS) to formulate a system architecture for pilots and air traffic control officers training in collaborative decision making by linking and integrating the virtual counterparts of flights and air traffic control operations. Collaborative decision-making training and the corresponding intelligent automation aids could be realised and supported. A performance analysis via a flight task undertaken with different computational load settings was prepared to evaluate the platform’s latency and integrity. The latency is presented using its 95% confidence interval, and integrity is presented using the percentage of data loss during wireless transmission. The results demonstrated convincing performance and a promising system robustness in both domains.
Collapse
|