Effects of the audiovisual conflict on auditory early processes

A Neuroergonomics Approach to Mental Workload, Engagement and Human Performance

The assessment and prediction of cognitive performance is a key issue for any discipline concerned with human operators in the context of safety-critical behavior. Most of the research has focused on the measurement of mental workload but this construct remains difficult to operationalize despite decades of research on the topic. Recent advances in Neuroergonomics have expanded our understanding of neurocognitive processes across different operational domains. We provide a framework to disentangle those neural mechanisms that underpin the relationship between task demand, arousal, mental workload and human performance. This approach advocates targeting those specific mental states that precede a reduction of performance efficacy. A number of undesirable neurocognitive states (mind wandering, effort withdrawal, perseveration, inattentional phenomena) are identified and mapped within a two-dimensional conceptual space encompassing task engagement and arousal. We argue that monitoring the prefrontal cortex and its deactivation can index a generic shift from a nominal operational state to an impaired one where performance is likely to degrade. Neurophysiological, physiological and behavioral markers that specifically account for these states are identified. We then propose a typology of neuroadaptive countermeasures to mitigate these undesirable mental states.

Inattentional deafness to auditory alarms: Inter-individual differences, electrophysiological signature and single trial classification

Inattentional deafness can have deleterious consequences in complex real-life situations (e.g. healthcare, aviation) leading to miss critical auditory signals. Such failure of auditory attention is thought to rely on top-down biasing mechanisms at the central executive level. A complementary approach to account for this phenomenon is to consider the existence of visual dominance over hearing that could be implemented via direct visual-to-auditory pathways. To investigate this phenomenon, thirteen aircraft pilots, equipped with a 32-channel EEG system, faced a low and high workload scenarii along with an auditory oddball task in a motion flight simulator. Prior to the flying task, the pilots were screened to assess their working memory span and visual dominance susceptibility. The behavioral results disclosed that the volunteers missed 57.7% of the auditory alarms in the difficult condition. Among all evaluated capabilities, only the visual dominance index was predictive of the miss rate in the difficult scenario. These findings provide behavioral evidences that other early cross-modal competitive process than top down modulation process could account for inattentional deafness. The electrophysiological analyses showed that the miss over the hit alarms led to a significant amplitude reduction of early perceptual (N100) and late attentional (P3a and P3b) event-related potentials components. Eventually, we implemented an EEG-based processing pipeline to perform single-trial classification of inattentional deafness. The results indicate that this processing chain could be used in an ecological setting as it led to 72.2% mean accuracy to discriminate missed from hit auditory alarms.

Auditory Task Irrelevance: A Basis for Inattentional Deafness

Objective This study investigates the neural basis of inattentional deafness, which could result from task irrelevance in the auditory modality. Background Humans can fail to respond to auditory alarms under high workload situations. This failure, termed inattentional deafness, is often attributed to high workload in the visual modality, which reduces one's capacity for information processing. Besides this, our capacity for processing auditory information could also be selectively diminished if there is no obvious task relevance in the auditory channel. This could be another contributing factor given the rarity of auditory warnings. Method Forty-eight participants performed a visuomotor tracking task while auditory stimuli were presented: a frequent pure tone, an infrequent pure tone, and infrequent environmental sounds. Participants were required either to respond to the presentation of the infrequent pure tone (auditory task-relevant) or not (auditory task-irrelevant). We recorded and compared the event-related potentials (ERPs) that were generated by environmental sounds, which were always task-irrelevant for both groups. These ERPs served as an index for our participants' awareness of the task-irrelevant auditory scene. Results Manipulation of auditory task relevance influenced the brain's response to task-irrelevant environmental sounds. Specifically, the late novelty-P3 to irrelevant environmental sounds, which underlies working memory updating, was found to be selectively enhanced by auditory task relevance independent of visuomotor workload. Conclusion Task irrelevance in the auditory modality selectively reduces our brain's responses to unexpected and irrelevant sounds regardless of visuomotor workload. Application Presenting relevant auditory information more often could mitigate the risk of inattentional deafness.

Disruption in neural phase synchrony is related to identification of inattentional deafness in real-world setting

Individuals often have reduced ability to hear alarms in real world situations (e.g., anesthesia monitoring, flying airplanes) when attention is focused on another task, sometimes with devastating consequences. This phenomenon is called inattentional deafness and usually occurs under critical high workload conditions. It is difficult to simulate the critical nature of these tasks in the laboratory. In this study, dry electroencephalography is used to investigate inattentional deafness in real flight while piloting an airplane. The pilots participating in the experiment responded to audio alarms while experiencing critical high workload situations. It was found that missed relative to detected alarms were marked by reduced stimulus evoked phase synchrony in theta and alpha frequencies (6-14 Hz) from 120 to 230 ms poststimulus onset. Correlation of alarm detection performance with intertrial coherence measures of neural phase synchrony showed different frequency and time ranges for detected and missed alarms. These results are consistent with selective attentional processes actively disrupting oscillatory coherence in sensory networks not involved with the primary task (piloting in this case) under critical high load conditions. This hypothesis is corroborated by analyses of flight parameters showing greater maneuvering associated with difficult phases of flight occurring during missed alarms. Our results suggest modulation of neural oscillation is a general mechanism of attention utilizing enhancement of phase synchrony to sharpen alarm perception during successful divided attention, and disruption of phase synchrony in brain networks when attentional demands of the primary task are great, such as in the case of inattentional deafness.

Neural signature of inattentional deafness

Inattentional deafness is the failure to hear otherwise audible sounds (usually alarms) that may occur under high workload conditions. One potential cause for its occurrence could be an attentional bottleneck that occurs when task demands are high, resulting in lack of resources for processing of additional tasks. In this fMRI experiment, we explore the brain regions active during the occurrence of inattentional deafness using a difficult perceptual-motor task in which the participants fly through a simulated Red Bull air race course and at the same time push a button on the joystick to the presence of audio alarms. Participants were instructed to focus on the difficult piloting task and to press the button on the joystick quickly when they noticed an audio alarm. The fMRI results revealed that audio misses relative to hits had significantly greater activity in the right inferior frontal gyrus IFG and the superior medial frontal cortex. Consistent with an attentional bottleneck, activity in these regions was also present for poor flying performance (contrast of gates missed versus gates passed for the flying task). A psychophysiological interaction analysis from the IFG identified reduced effective connectivity to auditory processing regions in the right superior temporal gyrus for missed audio alarms relative to audio alarms that were heard. This study identifies a neural signature of inattentional deafness in an ecologically valid situation by directly measuring differences in brain activity and effective connectivity between audio alarms that were not heard compared to those that were heard. Hum Brain Mapp 38:5440-5455, 2017. © 2017 Wiley Periodicals, Inc.

High Working Memory Load Impairs Language Processing during a Simulated Piloting Task: An ERP and Pupillometry Study

Given the important amount of visual and auditory linguistic information that pilots have to process, operating an aircraft generates a high working-memory load (WML). In this context, the ability to focus attention on relevant information and to remain responsive to concurrent stimuli might be altered. Consequently, understanding the effects of WML on the processing of both linguistic targets and distractors is of particular interest in the study of pilot performance. In the present work, participants performed a simplified piloting task in which they had to follow one of three colored aircraft, according to specific written instructions (i.e., the written word for the color corresponding to the color of one of the aircraft) and to ignore either congruent or incongruent concurrent auditory distractors (i.e., a spoken name of color). The WML was manipulated with an n-back sub-task. Participants were instructed to apply the current written instruction in the low WML condition, and the 2-back written instruction in the high WML condition. Electrophysiological results revealed a major effect of WML at behavioral (i.e., decline of piloting performance), electrophysiological, and autonomic levels (i.e., greater pupil diameter). Increased WML consumed resources that could not be allocated to the processing of the linguistic stimuli, as indexed by lower P300/P600 amplitudes. Also, significantly, lower P600 responses were measured in incongruent vs. congruent trials in the low WML condition, showing a higher difficulty reorienting attention toward the written instruction, but this effect was canceled in the high WML condition. This suppression of interference in the high load condition is in line with the engagement/distraction trade-off model. We propose that P300/P600 components could be reliable indicators of WML and that they allow an estimation of its impact on the processing of linguistic stimuli.

P300 event-related potential as an indicator of inattentional deafness?

An analysis of airplane accidents reveals that pilots sometimes purely fail to react to critical auditory alerts. This inability of an auditory stimulus to reach consciousness has been coined under the term of inattentional deafness. Recent data from literature tends to show that tasks involving high cognitive load consume most of the attentional capacities, leaving little or none remaining for processing any unexpected information. In addition, there is a growing body of evidence for a shared attentional capacity between vision and hearing. In this context, the abundant information in modern cockpits is likely to produce inattentional deafness. We investigated this hypothesis by combining electroencephalographic (EEG) measurements with an ecological aviation task performed under contextual variation of the cognitive load (high or low), including an alarm detection task. Two different audio tones were played: standard tones and deviant tones. Participants were instructed to ignore standard tones and to report deviant tones using a response pad. More than 31% of the deviant tones were not detected in the high load condition. Analysis of the EEG measurements showed a drastic diminution of the auditory P300 amplitude concomitant with this behavioral effect, whereas the N100 component was not affected. We suggest that these behavioral and electrophysiological results provide new insights on explaining the trend of pilots' failure to react to critical auditory information. Relevant applications concern prevention of alarms omission, mental workload measurements and enhanced warning designs.

Classification of single-trial auditory events using dry-wireless EEG during real and motion simulated flight

Application of neuro-augmentation technology based on dry-wireless EEG may be considerably beneficial for aviation and space operations because of the inherent dangers involved. In this study we evaluate classification performance of perceptual events using a dry-wireless EEG system during motion platform based flight simulation and actual flight in an open cockpit biplane to determine if the system can be used in the presence of considerable environmental and physiological artifacts. A passive task involving 200 random auditory presentations of a chirp sound was used for evaluation. The advantage of this auditory task is that it does not interfere with the perceptual motor processes involved with piloting the plane. Classification was based on identifying the presentation of a chirp sound vs. silent periods. Evaluation of Independent component analysis (ICA) and Kalman filtering to enhance classification performance by extracting brain activity related to the auditory event from other non-task related brain activity and artifacts was assessed. The results of permutation testing revealed that single trial classification of presence or absence of an auditory event was significantly above chance for all conditions on a novel test set. The best performance could be achieved with both ICA and Kalman filtering relative to no processing: Platform Off (83.4% vs. 78.3%), Platform On (73.1% vs. 71.6%), Biplane Engine Off (81.1% vs. 77.4%), and Biplane Engine On (79.2% vs. 66.1%). This experiment demonstrates that dry-wireless EEG can be used in environments with considerable vibration, wind, acoustic noise, and physiological artifacts and achieve good single trial classification performance that is necessary for future successful application of neuro-augmentation technology based on brain-machine interfaces.