Mapping brain activity during loss of situation awareness: an EEG investigation of a basis for top-down influence on perception

An Eye-Fixation Related Electroencephalography Technique for Predicting Situation Awareness: Implications for Driver State Monitoring Systems

OBJECTIVE

This study developed a fixation-related electroencephalography band power (FRBP) approach for situation awareness (SA) assessment in automated driving.

BACKGROUND

Maintaining good SA in Level 3 automated vehicles is crucial to drivers' takeover performance when the automated system fails. A multimodal fusion approach that enables the analysis of the visual behavioral and cognitive processes of SA can facilitate real-time assessment of SA in future driver state monitoring systems.

METHOD

Thirty participants performed three simulated automated driving tasks. After each task, the Situation Awareness Global Assessment Technique (SAGAT) was deployed to capture their SA about key elements that could affect their takeover task performance. Participants eye movements and brain activities were recorded. Data on their brain activity after each eye fixation on the key elements were extracted and labeled according to the correctness of the SAGAT. Mixed-effects models were used to identify brain regions that were indicative of SA, and machine learning models for SA assessment were developed based on the identified brain regions.

RESULTS

Participants' alpha and theta oscillation at frontal and temporal areas are indicative of SA. In addition, the FRBP technique can be used to predict drivers' SA with an accuracy of 88% using a neural network model.

CONCLUSION

The FRBP technique, which incorporates eye movements and brain activities, can provide more comprehensive evaluation of SA. Findings highlight the potential of utilizing FRBP to monitor drivers' SA in real-time.

APPLICATION

The proposed framework can be expanded and applied to driver state monitoring systems to measure human SA in real-world driving.

EEG and fNIRS are associated with situation awareness (hazard) prediction during a driving task

Situation awareness (SA) is important in many demanding tasks (e.g. driving). Assessing SA during training can indicate whether someone is ready to perform in the real world. SA is typically assessed by interrupting the task to ask questions about the situation or asking questions after task completion, assessing only momentary SA. An objective and continuous means of detecting SA is needed. We examined whether neurophysiological sensors are useful to objectively measure Level 3 SA (projection of events into the future) during a driving task. We measured SA by the speed at which participants responded to SA questions and the accuracy of responses. For EEG, beta and theta power were most sensitive to SA response time. For fNIRS, oxygenated haemoglobin (HbO) was most sensitive to accuracy. This is the first evidence to our knowledge that neurophysiological measures are useful for assessing Level 3 SA during an ecologically valid task.

Evidence of elevated situational awareness for active duty soldiers during navigation of a virtual environment

U.S. service members maintain constant situational awareness (SA) due to training and experience operating in dynamic and complex environments. Work examining how military experience impacts SA during visual search of a complex naturalistic environment, is limited. Here, we compare Active Duty service members and Civilians' physiological behavior during a navigational visual search task in an open-world virtual environment (VE) while cognitive load was manipulated. We measured eye-tracking and electroencephalogram (EEG) outcomes from Active Duty (N = 21) and Civilians (N = 15) while they navigated a desktop VE at a self-regulated pace. Participants searched and counted targets (N = 15) presented among distractors, while cognitive load was manipulated with an auditory Math Task. Results showed Active Duty participants reported significantly greater/closer to the correct number of targets compared to Civilians. Overall, Active Duty participants scanned the VE with faster peak saccade velocities and greater average saccade magnitudes compared to Civilians. Convolutional Neural Network (CNN) response (EEG P-300) was significantly weighted more to initial fixations for the Active Duty group, showing reduced attentional resources on object refixations compared to Civilians. There were no group differences in fixation outcomes or overall CNN response when comparing targets versus distractor objects. When cognitive load was manipulated, only Civilians significantly decreased their average dwell time on each object and the Active Duty group had significantly fewer numbers of correct answers on the Math Task. Overall, the Active Duty group explored the VE with increased scanning speed and distance and reduced cognitive re-processing on objects, employing a different, perhaps expert, visual search strategy indicative of increased SA. The Active Duty group maintained SA in the main visual search task and did not appear to shift focus to the secondary Math Task. Future work could compare how a stress inducing environment impacts these groups' physiological or cognitive markers and performance for these groups.

Recognizing the situation awareness of forklift operators based on EEG techniques in a field experiment

Lack of situation awareness (SA) is the primary cause of human errors when operating forklifts, so determining the SA level of the forklift operator is crucial to the safety of forklift operations. An EEG recognition approach of forklift operator SA in actual settings was presented in order to address the issues with invasiveness, subjectivity, and intermittency of existing measuring methods. In this paper, we conducted a field experiment that mimicked a typical forklift operation scenario to verify the differences in EEG states of forklift operators with different SA levels and investigate the correlation of multi-band combination features of each brain region of forklift operators with SA. Based on the sensitive EEG combination indexes, Support Vector Mechanism was used to construct a forklift operator SA recognition model. The results revealed that there were differences between forklift operators with high and low SA in the θ, α, and β frequency bands in zones F, C, P, and O; combined EEG indicators θ/β, (α + θ)/(α + β), and θ/(α + β) in zones F, P, and C were significantly correlated with SA; the recognition accuracy of the model reached 88.64% in the case of combined EEG indicators of zones C & F & P as input. It could provide a reference for SA measurement, contributing to the improvement of SA.

Physiological Measurements of Situation Awareness: A Systematic Review

OBJECTIVE

The goal of this systematic literature review is to investigate the relationship between indirect physiological measurements and direct measures of situation awareness (SA).

BACKGROUND

Across different environments and tasks, assessments of SA are often performed using techniques designed specifically to directly measure SA, such as SAGAT, SPAM, and/or SART. However, research suggests that indirect physiological sensing methods may also be capable of predicting SA. Currently, it is unclear which particular physiological approaches are sensitive to changes in SA.

METHOD

Seven databases were searched using the PRISMA reporting guidelines. Eligibility criteria included human-subject experiments that used at least one direct SA assessment technique, as well as at least one physiological measurement. Information extracted from each article was the physiological metric(s), the direct SA measurement(s), the correlation between these two metrics, and the experimental task(s). All studies underwent a quality assessment.

RESULTS

Twenty-five articles were included in this review. Eye tracking techniques were the most commonly used physiological measures, and correlations between conscious aspects of eye movement measures and direct SA scores were observed. Evidence for cardiovascular predictors of SA were mixed. EEG studies were too few to form strong conclusions, but were consistently positive.

CONCLUSION

Further investigation is needed to methodically collect more relevant data and comprehensively model the relationships between a wider range of physiological measurements and direct assessments of SA.

APPLICATION

This review will guide researchers and practitioners in methods to indirectly assess SA with sensors and highlight opportunities for future research on wearables and SA.

Using EEG to investigate the influence of boredom on prospective memory in top-down and bottom-up processing mechanisms for intelligent interaction

We aimed to investigate the alpha (α) activity in operators experiencing boredom while performing prolonged monitoring and prospective memory tasks using different processing mechanisms. Fifty-four participants underwent electroencephalography (EEG) and were found to have poorer prospective memory performance under top-down conditions. Further, α power and synchronisation were higher during bottom-up than in top-down processes, revealing an inhibition effect of the former. Significant differences in brain regions and hemispheres were identified to distinguish different cognitive processes in both information-processing mechanisms. Thus, people are likely to cope with boredom differently in terms of top-down and bottom-up processes. Specifically, a higher attention level was reported during top-down processing, to mitigate the negative influences of boredom. Overall, this study provides EEG evidence which suggests that prospective memory can be enhanced in top-down processing during prolonged monitoring tasks by increasing the salience of cues.

The Influence of Emotion Induced by Accidents and Incidents on Pilots’ Situation Awareness

(1) Background: This study examines the differences in emotions induced by accidents and incidents as emotional stimuli and the effects on pilot situation awareness (SA) after induction. (2) Method: Forty-five jet pilots were randomly divided into three groups after which their emotions were induced using the pictures of accident, incident, and neutral stimulus, respectively. (3) Results: The conditions of accidents and incidents both induced changes in the pilots’ happiness and sadness and the changes in the emotion were regulated by the emotional intelligence of pilots in the high SA group. The emotion induction, which caused a direct change in pilot’s happiness and fear, resulted in conditions that indirectly affected level 1 of SA in pilots. (4) Conclusions: The research elucidates the difference between accident and incident in inducing pilot emotions, and reminds us that SA level exerts the regulating effects on the same emotional induction conditions.

Mental stress and safety awareness during human-robot collaboration - Review

Human-robot collaboration (HRC) is an emerging research area that has gained tremendous attention in both academia and industry. Yet, the feature that humans and robots sharing the workplace has led to safety concerns. In particular, the mental stress or safety awareness of human teammates during HRC remains unclear but is also of great importance to workplace safety. In this manuscript, we reviewed twenty-five studies for understanding the relationships between HRC and workers' mental stress or safety awareness. Specifically, we aimed to understand: (1) robot-related factors that may affect human workers' mental stress or safety awareness, (2) a number of measurements that could be used to evaluate workers' mental stress in HRC, and (3) various methods for measuring safety awareness that had been adopted or could be applied in HRC. According to our literature review, robot-related factors including robot characteristics, social touching and trajectory have relationships with workers' mental stress or safety awareness. For the measurement of mental stress and safety awareness, each method mentioned has its validity and rationality. Additionally, a discussion related to the potential co-robot actions to lower mental stress or improve safety awareness as well as future implications were provided.

Brain augmentation and neuroscience technologies: current applications, challenges, ethics and future prospects

Ever since the dawn of antiquity, people have strived to improve their cognitive abilities. From the advent of the wheel to the development of artificial intelligence, technology has had a profound leverage on civilization. Cognitive enhancement or augmentation of brain functions has become a trending topic both in academic and public debates in improving physical and mental abilities. The last years have seen a plethora of suggestions for boosting cognitive functions and biochemical, physical, and behavioral strategies are being explored in the field of cognitive enhancement. Despite expansion of behavioral and biochemical approaches, various physical strategies are known to boost mental abilities in diseased and healthy individuals. Clinical applications of neuroscience technologies offer alternatives to pharmaceutical approaches and devices for diseases that have been fatal, so far. Importantly, the distinctive aspect of these technologies, which shapes their existing and anticipated participation in brain augmentations, is used to compare and contrast them. As a preview of the next two decades of progress in brain augmentation, this article presents a plausible estimation of the many neuroscience technologies, their virtues, demerits, and applications. The review also focuses on the ethical implications and challenges linked to modern neuroscientific technology. There are times when it looks as if ethics discussions are more concerned with the hypothetical than with the factual. We conclude by providing recommendations for potential future studies and development areas, taking into account future advancements in neuroscience innovation for brain enhancement, analyzing historical patterns, considering neuroethics and looking at other related forecasts.

Effects of Neural Mechanisms of Pretask Resting EEG Alpha Information on Situational Awareness: A Functional Connectivity Approach

OBJECTIVE

In this study, the influence of pretask resting neural mechanisms on situational awareness (SA)-task is studied.

BACKGROUND

Pretask electroencephalography (EEG) information and Stroop effect are known to influence task engagement independently. However, neural mechanisms of pretask resting absolute alpha (PRAA) and pretask resting alpha frontal asymmetry (PRAFA) in influencing SA-task which is undergoing Stroop effect is still not understood.

METHOD

The study involved pretask resting EEG measurements from 18 healthy individuals followed by functional magnetic resonance imaging (fMRI) acquisition during SA-task. To understand the effect of pretask alpha information and Stroop effect on SA, a robust correlation between mean reaction time, SA Index, PRAA, and PRAFA were assessed. Furthermore, neural underpinnings of PRAA, PRAFA in SA-task, and functional connectivity were analyzed through the EEG-informed fMRI approach.

RESULTS

Significant robust correlation of reaction time was observed with SA Index (Pearson: r = .50, pcorr = .05) and PRAFA (Pearson: r = .63; pcorr = .01), respectively. Similarly, SA Index significantly correlated with PRAFA (Pearson: r = .56, pcorr = .01; Spearman: r = .61, pcorr = .007), and PRAA (Pearson: r = .59, pcorr = .005; Spearman: r = .59, pcorr = .002). Neural underpinnings of SA-task revealed regions involved in visual-processing and higher-order cognition. PRAA was primarily underpinned at frontal-temporal areas and functionally connected to SA-task regions pertaining to the emotional regulation. PRAFA has correlated with limbic and parietal regions, which are involved in integration of visual, emotion, and memory information of SA-task.

CONCLUSION

The results suggest a strong association of reaction time with SA-task and PRAFA and strongly support the hypothesis that PRAFA, PRAA, and associated neural mechanisms significantly influence the outcome of SA-task.

APPLICATION

It is beneficial to study the effect of pretask resting information on SA-task to improve SA.

Investigating the relationship between eye movements and situation awareness in weather forecasting

Physiological indicators, including eye tracking measures, may provide insight into human decision making and cognition in many domains, including weather forecasting. Situation awareness (SA), a critical component of forecast decision making, is commonly conceptualized as the degree to which information is perceived, understood, and projected into a future context. Drawing upon recent applications of eye tracking in the study of forecaster decision making, we investigate the relationship among eye movement measures, automation, and SA assessed through a freeze probe assessment method. In addition, we explore the relationship between an automated forecasting decision aid use and information seeking behavior. In this study, a sample of professional weather forecasters completed a series of tasks, informed by a set of forecasting decision aids, and with variable access to an experimental automated tool, while an eye tracking system captured data related to eye movements and information usage. At the end of each forecasting task, participants responded to a set of questions related to the environmental situation in the framework of a survey-based assessment technique in order to assess their level of situation awareness. Regression analysis revealed a moderate relationship between the SA measure and eye tracking metrics, supporting the hypothesis that eye tracking may have utility in assessing SA. The results support the use of eye tracking in the assessment of specific and measurable attributes of the decision-making process in weather forecasting. The findings are discussed in light of potential benefits that eye tracking could bring to human performance assessment as well as decision-making research in the forecasting domain.

Neurotechnologies for Human Cognitive Augmentation: Current State of the Art and Future Prospects

Recent advances in neuroscience have paved the way to innovative applications that cognitively augment and enhance humans in a variety of contexts. This paper aims at providing a snapshot of the current state of the art and a motivated forecast of the most likely developments in the next two decades. Firstly, we survey the main neuroscience technologies for both observing and influencing brain activity, which are necessary ingredients for human cognitive augmentation. We also compare and contrast such technologies, as their individual characteristics (e.g., spatio-temporal resolution, invasiveness, portability, energy requirements, and cost) influence their current and future role in human cognitive augmentation. Secondly, we chart the state of the art on neurotechnologies for human cognitive augmentation, keeping an eye both on the applications that already exist and those that are emerging or are likely to emerge in the next two decades. Particularly, we consider applications in the areas of communication, cognitive enhancement, memory, attention monitoring/enhancement, situation awareness and complex problem solving, and we look at what fraction of the population might benefit from such technologies and at the demands they impose in terms of user training. Thirdly, we briefly review the ethical issues associated with current neuroscience technologies. These are important because they may differentially influence both present and future research on (and adoption of) neurotechnologies for human cognitive augmentation: an inferior technology with no significant ethical issues may thrive while a superior technology causing widespread ethical concerns may end up being outlawed. Finally, based on the lessons learned in our analysis, using past trends and considering other related forecasts, we attempt to forecast the most likely future developments of neuroscience technology for human cognitive augmentation and provide informed recommendations for promising future research and exploitation avenues.

Quantitative Analysis of Situation Awareness (QASA): modelling and measuring situation awareness using signal detection theory

This paper presents a model of situation awareness (SA) that emphasises that SA is necessarily built using a subset of available information. A technique (Quantitative Analysis of Situation Awareness - QASA), based around signal detection theory, has been developed from this model that provides separate measures of actual SA (ASA) and perceived SA (PSA), together with a feature unique to QASA, a measure of bias (information acceptance). These measures allow the exploration of the relationship between actual SA, perceived SA and information acceptance. QASA can also be used for the measurement of dynamic ASA, PSA and bias. Example studies are presented and full details of the implementation of the QASA technique are provided. Practitioner Summary: This paper presents a new model of situation awareness (SA) together with an associated tool (Quantitative Analysis of Situation Awareness - QASA) that employs signal detection theory to measure several aspects of SA, including actual and perceived SA and information acceptance. Full details are given of the implementation of the tool.