The Role of Cognitive and Perceptual Loads in Inattentional Deafness

Physiological measures of operators' mental state in supervisory process control tasks: a scoping review

Physiological measures are often used to assess the mental state of human operators in supervisory process control tasks. However, the diversity of research approaches creates a heterogeneous landscape of empirical evidence. To map existing evidence and provide guidance to researchers and practitioners, this paper systematically reviews 109 empirical studies that report relationships between peripheral nervous system measures and mental state dimensions (e.g. mental workload, mental fatigue, stress, and vigilance) of interest. Ocular and electrocardiac measures were the most prominent measures across application fields. Most studies sought to validate such measures for reliable assessments of cognitive task demands and time on task, with measures of pupil size receiving the most empirical support. In comparison, less research examined the utility of physiological measures in predicting human task performance. This approach is discussed as an opportunity to focus on operators' individual response to cognitive task demands and to advance the state of research.

Recognition and evaluation of mental workload in different stages of perceptual and cognitive information processing using a multimodal approach

This study explores the effects of different perceptual and cognitive information processing stages on mental workload by assessing multimodal indicators of mental workload such as the NASA-TLX, task performance, ERPs and eye movements. Repeated measures ANOVA of the data showed that among ERP indicators, P1, N1 and N2 amplitudes were sensitive to perceptual load (P-load), P3 amplitude was sensitive to P-load only in the prefrontal region during high cognitive load (C-load) states, and P3 amplitude in the occipital and parietal regions was sensitive to C-load. Among the eye movement indicators, blink frequency was sensitive to P-load in all C-load states, but to C-load in only low P-load states; pupil diameter and blink duration were sensitive to both P-load and C-load. Based on the above indicators, the k-nearest neighbours (KNN) algorithm was used to propose a classification method for the four different mental workload states with an accuracy of 97.89%.Practitioner summary: Based on the results of this study, it is possible to implement the monitoring of mental workload states and optimise brain task allocation in operations involving high mental workload, such as human-computer interaction.

Fast & scrupulous: Gesture-based alarms improve accuracy and reaction times under various mental workload levels. An ERSP study

In high-risk environments, fast and accurate responses to warning systems are essential to efficiently handle emergency situations. The aim of the present study was twofold: 1) investigating whether hand action videos (i.e., gesture alarms) trigger faster and more accurate responses than text alarm messages (i.e., written alarms), especially when mental workload (MWL) is high; and 2) investigating the brain activity in response to both types of alarms as a function of MWL. Regardless of MWL, participants (N = 28) were found to be both faster and more accurate when responding to gesture alarms than to written alarms. Brain electrophysiological results suggest that this greater efficiency might be due to a facilitation of the action execution, reflected by the decrease in mu and beta power observed around the response time window observed at C3 and C4 electrodes. These results suggest that gesture alarms may improve operators' performances in emergency situations.

The effect of working memory load on inattentional deafness during aeronautical decision-making

Operating an aircraft requires pilots to handle a significant amount of multi-modal information, which creates a high working memory load. Detecting auditory alarms in this high-load scenario is crucial for aviation safety. According to cognitive control load theory, an increase in working memory load may enhance distractor interference, resulting in improved detection sensitivity for task-irrelevant stimuli. Therefore, understanding the effect of working memory load on auditory alarm detection is of particular interest in aviation safety research. The studies were designed to investigate the effect of storage load and executive function load of working memory on auditory alarm detection during aeronautical decision-making through three experiments. In Experiment 1 and 2, participants performed an aeronautical decision-making task while also detecting an auditory alarm during the retention interval of a working memory task (visual-spatial, visual-verbal and auditory-verbal). In Experiment 3, participants were required to detect an auditory alarm while performing the 2-back and 3-back aeronautical decision-making tasks. Experiment 1 found that the auditory alarm sensitivity was higher in conditions of low visual-spatial working memory storage load compare to high load conditions. Experiment 2 found that a high storage load of visual-verbal working memory reduced auditory alarm sensitivity but auditory-verbal working memory load did not. Experiment 3 found that, unlike storage load, auditory alarm sensitivity was stronger under high executive function load relative to low executive function load. These findings show that working memory storage load and executive function load have different effects on auditory alarm sensitivity. The relationship between executive function and auditory alarm sensitivity supports cognitive control load theory, while the impact of the storage function on auditory alarm sensitivity does not adhere to this theory.

The unnoticed zoo: Inattentional deafness to animal sounds in music

Inattentional unawareness potentially occurs in several different sensory domains but is mainly described in visual paradigms ("inattentional blindness"; e.g., Simons & Chabris, 1999, Perception, 28, 1059-1074). Dalton and Fraenkel (2012, Cognition, 124, 367-372) were introducing "inattentional deafness" by showing that participants missed by 70% a voice repeatedly saying "I'm a Gorilla" when focusing on a primary conversation. The present study expanded this finding from the acoustic domain in a multifaceted way: First, we extended the validity perspective by using 10 acoustic samples-specifically, excerpts of popular musical pieces from different music genres. Second, we used as the secondary acoustic signal animal sounds. Those sounds originate from a completely different acoustic domain and are therefore highly distinctive from the primary sound. Participants' task was to count different musical features. Results (N = 37 participants) showed that the frequency of missed animal sounds was higher in participants with higher attentional focus and motivation. Additionally, attentional focus, perceptual load, and feature similarity/saliency were analyzed and did not have an influence on detecting or missing animal sounds. We could demonstrate that for 31.2% of the music plays, people did not recognize highly salient animal voices (regarding the type of acoustic source as well as the frequency spectra) when executing the primary (counting) task. This uncovered, significant effect supports the idea that inattentional deafness is even available when the unattended acoustic stimuli are highly salient.

Classification of Electrophysiological Signatures With Explainable Artificial Intelligence: The Case of Alarm Detection in Flight Simulator

Relevant sounds such as alarms are sometimes involuntarily ignored, a phenomenon called inattentional deafness. This phenomenon occurs under specific conditions including high workload (i.e., multitasking) and/or cognitive fatigue. In the context of aviation, such an error can have drastic consequences on flight safety. This study uses an oddball paradigm in which participants had to detect rare sounds in an ecological context of simulated flight. Cognitive fatigue and cognitive load were manipulated to trigger inattentional deafness, and brain activity was recorded via electroencephalography (EEG). Our results showed that alarm omission and alarm detection can be classified based on time-frequency analysis of brain activity. We reached a maximum accuracy of 76.4% when the algorithm was trained on all participants and a maximum of 90.5%, on one participant, when the algorithm was trained individually. This method can benefit from explainable artificial intelligence to develop efficient and understandable passive brain-computer interfaces, improve flight safety by detecting such attentional failures in real time, and give appropriate feedback to pilots, according to our ambitious goal, providing them with reliable and rich human/machine interactions.

Assessing Perceptual Load and Cognitive Load by Fixation-Related Information of Eye Movements

Assessing mental workload is imperative for avoiding unintended negative consequences in critical situations such as driving and piloting. To evaluate mental workload, measures of eye movements have been adopted, but unequivocal results remain elusive, especially those related to fixation-related parameters. We aimed to resolve the discrepancy of previous results by differentiating two kinds of mental workload (perceptual load and cognitive load) and manipulated them independently using a modified video game. We found opposite effects of the two kinds of mental workload on fixation-related parameters: shorter fixation durations and more fixations when participants played an episode with high (vs. low) perceptual load, and longer fixation durations and fewer fixations when they played an episode with high (vs. low) cognitive load. Such opposite effects were in line with the load theory and demonstrated that fixation-related parameters can be used to index mental workload at different (perceptual and cognitive) stages of mental processing.

High expectancy influences the role of cognitive load in inattentional deafness during landing decision-making

Neglecting a critical auditory alarm is a major obstacle to maintaining a safe environment, especially in aviation. Earlier studies have indicated that tasks with a higher perceptual or cognitive load in the visual modality influence the processing of auditory stimuli. It is unclear, however, whether other factors, such as memory failure, active neglect, or expectancy influence the effect of cognitive load on auditory alarm detection sensitivity during aeronautical decision-making. In this study, we investigated this issue in three laboratory experiments using the technique of signal detection analysis, in which participants were asked to make a landing decision based on indicators of the instrument landing system while also trying to detect an audible alarm. We found that the sensitivity of auditory alarm detection was reduced under conditions of high cognitive load and that this effect persisted even when the auditory detection response occurred first (before the landing decision response) and when the probability of an auditory alarm was 40%. However, the sensitivity of auditory detection was not influenced by cognitive load under high expectancy conditions (60% probability of alarm presentation). Furthermore, the value of the response bias was reduced under high cognitive load conditions when the probability of an auditory alarm was low (20%). With an increase in the level of expectancy (40% and 60% probability of alarm presentation), it was found that cognitive load did not influence the response bias. These findings indicate that visual cognitive load affects the sensitivity to an auditory alarm only at a low expectancy level (20% and 40% probability of alarm presentation). The effect of cognitive load on the sensitivity to an auditory alarm was not due to memory failure or active neglect and the response bias was more sensitive to the expectancy factor.

Cross-Task Consistency of Electroencephalography-Based Mental Workload Indicators: Comparisons Between Power Spectral Density and Task-Irrelevant Auditory Event-Related Potentials

Mental workload (MWL) estimators based on ongoing electroencephalography (EEG) and event-related potentials (ERPs) have shown great potentials to build adaptive aiding systems for human-machine systems by estimating MWL in real time. However, extracting EEG features which are consistent in indicating MWL across different tasks is still one of the critical challenges. This study attempts to compare the cross-task consistency in indexing MWL variations between two commonly used EEG-based MWL indicators, power spectral density (PSD) of ongoing EEG and task-irrelevant auditory ERPs (tir-aERPs). The verbal N-back and the multi-attribute task battery (MATB), both with two difficulty levels, were employed in the experiment, along with task-irrelevant auditory probes. EEG was recorded from 17 subjects when they were performing the tasks. The tir-aERPs elicited by the auditory probes and the relative PSDs of ongoing EEG between two consecutive auditory probes were extracted and statistically analyzed to reveal the effects of MWL and task type. Discriminant analysis and support vector machine were employed to examine the generalization of tir-aERP and PSD features in indexing MWL variations across different tasks. The results showed that the amplitudes of tir-aERP components, N1, early P3a, late P3a, and the reorienting negativity, significantly decreased with the increasing MWL in both N-back and MATB. Task type had no obvious influence on the amplitudes and topological layout of the MWL-sensitive tir-aERP features. The relative PSDs in θ, α, and low β bands were also sensitive to MWL variations. However, the MWL-sensitive PSD features and their topological patterns were significantly affected by task type. The cross-task classification results based on tir-aERP features also significantly outperformed the PSD features. These results suggest that the tir-aERPs should be potentially more consistent MWL indicators across very different task types when compared to PSD. The current study may provide new insights to our understanding of the common and distinctive neuropsychological essences of MWL across different tasks.

When two is worse than one: The deleterious impact of multisensory stimulation on response inhibition

Multisensory facilitation is known to improve the perceptual performances and reaction times of participants in a wide range of tasks, from detection and discrimination to memorization. We asked whether a multimodal signal can similarly improve action inhibition using the stop-signal paradigm. Indeed, consistent with a crossmodal redundant signal effect that relies on multisensory neuronal integration, the threshold for initiating behavioral responses is known for being reached faster with multisensory stimuli. To evaluate whether this phenomenon also occurs for inhibition, we compared stop signals in unimodal (human faces or voices) versus audiovisual modalities in natural or degraded conditions. In contrast to the expected multisensory facilitation, we observed poorer inhibition efficiency in the audiovisual modality compared with the visual and auditory modalities. This result was corroborated by both response probabilities and stop-signal reaction times. The visual modality (faces) was the most effective. This is the first demonstration of an audiovisual impairment in the domain of perception and action. It suggests that when individuals are engaged in a high-level decisional conflict, bimodal stimulation is not processed as a simple multisensory object improving the performance but is perceived as concurrent visual and auditory information. This absence of unity increases task demand and thus impairs the ability to revise the response.

Effects of automation reliability on error detection and attention to auditory stimuli in a multi-tasking environment

Automated aids are engineered to support operators' decision-making in complex and task-saturated environments, alerting them of system status and critical incidents. However, even the most advanced technologies are susceptible to failure. Monitoring imperfect automated systems poses unique challenges related to operator attention and workload. This study empirically examined the effect of unreliable automation on monitoring performance and responses to auditory stimuli in a multi-tasking environment. Participants completed an experimental trial consisting of four flight-related tasks while monitoring for automation failures at one of three levels of automation reliability. Participants in a high reliability condition responded more quickly and frequently to auditory messages. No performance differences were found in system monitoring performance between reliability conditions. These results are relevant to the design of automated system and delivery of automated alerts, and they have implications for operator attention allocation strategies in multi-tasking environments.

A Neuroergonomics Approach to Mental Workload, Engagement and Human Performance

The assessment and prediction of cognitive performance is a key issue for any discipline concerned with human operators in the context of safety-critical behavior. Most of the research has focused on the measurement of mental workload but this construct remains difficult to operationalize despite decades of research on the topic. Recent advances in Neuroergonomics have expanded our understanding of neurocognitive processes across different operational domains. We provide a framework to disentangle those neural mechanisms that underpin the relationship between task demand, arousal, mental workload and human performance. This approach advocates targeting those specific mental states that precede a reduction of performance efficacy. A number of undesirable neurocognitive states (mind wandering, effort withdrawal, perseveration, inattentional phenomena) are identified and mapped within a two-dimensional conceptual space encompassing task engagement and arousal. We argue that monitoring the prefrontal cortex and its deactivation can index a generic shift from a nominal operational state to an impaired one where performance is likely to degrade. Neurophysiological, physiological and behavioral markers that specifically account for these states are identified. We then propose a typology of neuroadaptive countermeasures to mitigate these undesirable mental states.

The impact of alerting designs on air traffic controller's eye movement patterns and situation awareness

This research investigated controller' situation awareness by comparing COOPANS's acoustic alerts with newly designed semantic alerts. The results demonstrate that ATCOs' visual scan patterns had significant differences between acoustic and semantic designs. ATCOs established different eye movement patterns on fixations number, fixation duration and saccade velocity. Effective decision support systems require human-centered design with effective stimuli to direct ATCO's attention to critical events. It is necessary to provide ATCOs with specific alerting information to reflect the nature of the critical situation in order to minimise the side effects of startle and inattentional deafness. Consequently, the design of a semantic alert can significantly reduce ATCOs' response time, therefore providing valuable extra time in a time-limited situation to formulate and execute resolution strategies in critical air safety events. The findings of this research indicate that the context-specified design of semantic alerts could improve ATCO's situational awareness and significantly reduce response time in the event of Short Term Conflict Alert (STCA) activation which alerts to two aircraft having less than the required lateral or vertical separation. Practitioner Summary: Eye movements are closely linked with visual attention and can be analysed to explore shifting attention whilst performing monitoring tasks. This research has found that context-specific designed semantic alerts facilitated improved ATCO cognitive processing by integrating visual and auditory resources. Semantic designs have been demonstrated to be superior to acoustic design by directing the operator's attention more quickly to critical situations.Abbreviations: APW: area proximity warning; ASRS: aviation safety reporting system; ATC: air traffic control; ATCO: air traffic controller; ATM: air traffic management; COOPANS: cooperation between air navigation service providers; HCI: human-computer interaction; IAA: irish aviation authority; MSAW: minimum safe altitude warning; MTCD: medium-term conflict detection; SA: situation awareness; STCA: short term conflict alert; TP: trajectory prediction.

Inattentional deafness to auditory alarms: Inter-individual differences, electrophysiological signature and single trial classification

Inattentional deafness can have deleterious consequences in complex real-life situations (e.g. healthcare, aviation) leading to miss critical auditory signals. Such failure of auditory attention is thought to rely on top-down biasing mechanisms at the central executive level. A complementary approach to account for this phenomenon is to consider the existence of visual dominance over hearing that could be implemented via direct visual-to-auditory pathways. To investigate this phenomenon, thirteen aircraft pilots, equipped with a 32-channel EEG system, faced a low and high workload scenarii along with an auditory oddball task in a motion flight simulator. Prior to the flying task, the pilots were screened to assess their working memory span and visual dominance susceptibility. The behavioral results disclosed that the volunteers missed 57.7% of the auditory alarms in the difficult condition. Among all evaluated capabilities, only the visual dominance index was predictive of the miss rate in the difficult scenario. These findings provide behavioral evidences that other early cross-modal competitive process than top down modulation process could account for inattentional deafness. The electrophysiological analyses showed that the miss over the hit alarms led to a significant amplitude reduction of early perceptual (N100) and late attentional (P3a and P3b) event-related potentials components. Eventually, we implemented an EEG-based processing pipeline to perform single-trial classification of inattentional deafness. The results indicate that this processing chain could be used in an ecological setting as it led to 72.2% mean accuracy to discriminate missed from hit auditory alarms.

Neural signature of inattentional deafness

Inattentional deafness is the failure to hear otherwise audible sounds (usually alarms) that may occur under high workload conditions. One potential cause for its occurrence could be an attentional bottleneck that occurs when task demands are high, resulting in lack of resources for processing of additional tasks. In this fMRI experiment, we explore the brain regions active during the occurrence of inattentional deafness using a difficult perceptual-motor task in which the participants fly through a simulated Red Bull air race course and at the same time push a button on the joystick to the presence of audio alarms. Participants were instructed to focus on the difficult piloting task and to press the button on the joystick quickly when they noticed an audio alarm. The fMRI results revealed that audio misses relative to hits had significantly greater activity in the right inferior frontal gyrus IFG and the superior medial frontal cortex. Consistent with an attentional bottleneck, activity in these regions was also present for poor flying performance (contrast of gates missed versus gates passed for the flying task). A psychophysiological interaction analysis from the IFG identified reduced effective connectivity to auditory processing regions in the right superior temporal gyrus for missed audio alarms relative to audio alarms that were heard. This study identifies a neural signature of inattentional deafness in an ecologically valid situation by directly measuring differences in brain activity and effective connectivity between audio alarms that were not heard compared to those that were heard. Hum Brain Mapp 38:5440-5455, 2017. © 2017 Wiley Periodicals, Inc.

EEG-Based Cognitive Control Behaviour Assessment: an Ecological study with Professional Air Traffic Controllers

Several models defining different types of cognitive human behaviour are available. For this work, we have selected the Skill, Rule and Knowledge (SRK) model proposed by Rasmussen in 1983. This model is currently broadly used in safety critical domains, such as the aviation. Nowadays, there are no tools able to assess at which level of cognitive control the operator is dealing with the considered task, that is if he/she is performing the task as an automated routine (skill level), as procedures-based activity (rule level), or as a problem-solving process (knowledge level). Several studies tried to model the SRK behaviours from a Human Factor perspective. Despite such studies, there are no evidences in which such behaviours have been evaluated from a neurophysiological point of view, for example, by considering brain activity variations across the different SRK levels. Therefore, the proposed study aimed to investigate the use of neurophysiological signals to assess the cognitive control behaviours accordingly to the SRK taxonomy. The results of the study, performed on 37 professional Air Traffic Controllers, demonstrated that specific brain features could characterize and discriminate the different SRK levels, therefore enabling an objective assessment of the degree of cognitive control behaviours in realistic settings.