Engaging the human operator: a review of the theoretical support for the vigilance decrement and a discussion of practical applications

Activation thresholds, not quitting thresholds, account for the low prevalence effect in dynamic search

The low-prevalence effect (LPE) is the finding that target detection rates decline as targets become less frequent in a visual search task. A major source of this effect is thought to be that fewer targets result in lower quitting thresholds, i.e., observers respond target-absent after looking at fewer items compared to searches with a higher prevalence of targets. However, a lower quitting threshold does not directly account for an LPE in searches where observers continuously monitor a dynamic display for targets. In these tasks there are no discrete "trials" to which a quitting threshold could be applied. This study examines whether the LPE persists in this type of dynamic search context. Experiment 1 was a 2 (dynamic/static) x 2 (10%/40% prevalence targets) design. Although overall performance was worse in the dynamic task, both tasks showed a similar magnitude LPE. In Experiment 2, we replicated this effect using a task where subjects searched for either of two targets (Ts and Ls). One target appeared infrequently (10%) and the other moderately (40%). Given this method of manipulating prevalence rate, the quitting threshold explanation does not account for the LPE even for static displays. However, replicating Experiment 1, we found an LPE of similar magnitude for both search scenarios, and lower target detection rates with the dynamic displays, demonstrating the LPE is a potential concern for both static and dynamic searches. These findings suggest an activation threshold explanation of the LPE may better account for our observations than the traditional quitting threshold model.

The effect of interstimulus interval on sustained attention

The ability of nervous systems to filter out irrelevant and repetitive stimuli may prevent animals from becoming 'saturated' with excess information. However, animals must be particular about which stimuli to attend to and which to ignore, as mistakes may be costly. Using a comparative approach, we explored the effect of interstimulus interval (ISI) between repeated presentations of visual stimuli presented on a screen to test the decrease in responses (response decrement) of both Trite planiceps jumping spiders and untrained Columba livia pigeons, animals with comparable visual ability despite having structurally different visual systems and brain size. We used ISIs of 2.5 s, 5 s, 10 s, predicting that decreases in ISI would lead to progressively less responses to the stimuli. Following from previous work on T. planiceps, we also manipulated pigeon hunger level, finding that hungry birds were initially more responsive than sated pigeons, but the rate of decrease in responses to the stimulus did not differ between the two groups. While a clear response decrement was seen in both species across all conditions, shorter ISIs resulted in more dramatic response decrements, aligning with previous work and with the resource depletion theory posited in the human-based literature.

Can transcutaneous auricular vagus nerve stimulation mitigate vigilance loss? Examining the effects of stimulation at individualized versus constant current intensity

According to the arousal model of vigilance, the locus coeruleus-norepinephrine (LC-NE) system modulates sustained attention over long periods by regulating physiological arousal. Recent research has proposed that transcutaneous auricular vagus nerve stimulation (taVNS) modulates indirect physiological markers of LC-NE activity, although its effects on vigilance have not yet been examined. Aiming to develop a safe and noninvasive procedure to prevent vigilance failures in prolonged tasks, the present study examined whether taVNS can mitigate vigilance loss while modulating indirect markers of LC-NE activity. Following a preregistered protocol (https://osf.io/tu2xy/), 50 participants completed three repeated sessions in a randomized order, in which either active taVNS at individualized intensity set by participant, active taVNS set at 0.5 mA for all participants, or sham taVNS, was delivered while performing an attentional and vigilance task (i.e., ANTI-Vea). Changes in salivary alpha-amylase and cortisol concentrations were measured as markers of LC-NE activity. Self-reports of feelings associated with stimulation and guessing rate of active/sham conditions supported the efficacy of the single-blind procedure. Contrary to our predictions, the observed vigilance decrement was not modulated by active taVNS. Pairwise comparisons showed a mitigation by active taVNS on cortisol reduction across time. Interestingly, Spearman's correlational analyses showed some interindividual effects of taVNS on indirect markers of LC-NE, evidenced by positive associations between changes in salivary alpha-amylase and cortisol in active but not sham taVNS. We highlight the relevance of replicating and extending the present outcomes, investigating further parameters of stimulation and its effects on other indirect markers of LC-NE activity.

Physiological measures of operators' mental state in supervisory process control tasks: a scoping review

Physiological measures are often used to assess the mental state of human operators in supervisory process control tasks. However, the diversity of research approaches creates a heterogeneous landscape of empirical evidence. To map existing evidence and provide guidance to researchers and practitioners, this paper systematically reviews 109 empirical studies that report relationships between peripheral nervous system measures and mental state dimensions (e.g. mental workload, mental fatigue, stress, and vigilance) of interest. Ocular and electrocardiac measures were the most prominent measures across application fields. Most studies sought to validate such measures for reliable assessments of cognitive task demands and time on task, with measures of pupil size receiving the most empirical support. In comparison, less research examined the utility of physiological measures in predicting human task performance. This approach is discussed as an opportunity to focus on operators' individual response to cognitive task demands and to advance the state of research.

The Limits of Clinician Vigilance as an AI Safety Bulwark

This Viewpoint examines the potential problems of clinician reliance on the use of artificial intelligence (AI) in health care and offers suggestions on how AI could be designed to promote clinician vigilance.

A video processing and machine learning based method for evaluating safety-critical operator engagement in a motorway control room

In safety-critical automatic systems, safety can be compromised if operators lack engagement. Effective detection of undesirable engagement states can inform the design of interventions for enhancing engagement. However, the existing engagement measurement methods suffer from several limitations which damage their effectiveness in the work environment. A novel engagement evaluation methodology, which adopts Artificial Intelligence (AI) technologies, has been proposed. It was developed using motorway control room operators as subjects. Openpose and Open Source Computer Vision Library (OpenCV) were used to estimate the body postures of operators, then a Support Vector Machine (SVM) was utilised to build the engagement evaluation model based on discrete states of operator engagement. The average accuracy of the evaluation results reached 0.89 and the weighted average precision, recall, and F1-score were all above 0.84. This study emphasises the importance of specific data labelling when measuring typical engagement states, forming the basis for potential control room improvements.Practitioner summary: This study demonstrates an automatic, real-time, objective, and relatively unobtrusive method for measuring dynamic operator engagement states. Computer vision technologies were used to estimate body posture, then machine learning (ML) was utilised to build the engagement evaluation model. The overall evaluation shows the effectiveness of this framework.Abbreviations: AI: Artificial Intelligence; OpenCV: Open Source Computer Vision Library; SVM: Support Vector Machine; UWES: Utrecht Work Engagement Scale; ISA Engagement Scale: Intellectual, Social, Affective Engagement Scale; DSSQ: Dundee Stress State Questionnaire; SSSQ: Short Stress State Questionnaire; EEG: electroencephalography; ECG: Electrocardiography; VMOE: Video-based Measurement for Operator Engagement; CMU: Carnegie Mellon University; CNN: Convolutional Neural Network; 2D: two dimensional; ML: Machine learning.

Psychometric curves reveal changes in bias, lapse rate, and guess rate in an online vigilance task

When human monitors are required to detect infrequent signals among noise, they typically exhibit a decline in correct detections over time. Researchers have attributed this vigilance decrement to three alternative mechanisms: shifts in response bias, losses of sensitivity, and attentional lapses. The current study examined the extent to which changes in these mechanisms contributed to the vigilance decrement in an online monitoring task. Participants in two experiments (N = 102, N = 192) completed an online signal detection task, judging whether the separation between two probes each trial exceeded a criterion value. Separation was varied across trials and data were fit with logistic psychometric curves using Bayesian hierarchical parameter estimation. Parameters representing sensitivity, response bias, attentional lapse rate, and guess rate were compared across the first and last 4 minutes of the vigil. Data gave decisive evidence of conservative bias shifts, an increased attentional lapse rate, and a decreased positive guess rate over time on task, but no strong evidence for or against an effect of sensitivity. Sensitivity decrements appear less robust than criterion shifts or attention lapses as causes of the vigilance loss.

Event-related potentials associated with attentional networks evidence changes in executive and arousal vigilance

Attention is regulated by three independent but interacting networks, that is, alerting, comprising phasic alertness and vigilance, orienting, and executive control. Previous studies analyzing event-related potentials (ERPs) associated with attentional networks have focused on phasic alertness, orienting, and executive control, without an independent measure of vigilance. ERPs associated with vigilance have been instead measured in separate studies and via different tasks. The present study aimed to differentiate ERPs associated with attentional networks by simultaneously measuring vigilance along with phasic alertness, orienting, and executive control. Forty participants (34 women, age: M = 25.96; SD = 4.96) completed two sessions wherein the electroencephalogram was recorded while they completed the Attentional Networks Test for Interactions and Vigilance-executive and arousal components, a task that measures phasic alertness, orienting, and executive control along with executive (i.e., detection of infrequent critical signals) and arousal (i.e., sustaining a fast reaction to environmental stimuli) vigilance. ERPs previously associated with attentional networks were replicated here: (a) N1, P2, and contingent negative variation for phasic alertness; (b) P1, N1, and P3 for orienting; and (c) N2 and slow positivity for executive control. Importantly, different ERPs were associated with vigilance: while the executive vigilance decrement was associated with an increase in P3 and slow positivity across time-on-task, arousal vigilance loss was associated with reduced N1 and P2 amplitude. The present study shows that attentional networks can be described by different ERPs simultaneously observed in a single session, including independent measures of executive and arousal vigilance on its assessment.

Time on task and task load in visual inspection: A four-month field study with X-ray baggage screeners

Previous studies suggest that performance in visual inspection and typical vigilance tasks depend on time on task and task load. European regulation mandates that security officers (screeners) take a break or change tasks after 20 min of X-ray baggage screening. However, longer screening durations could reduce staffing challenges. We investigated the effects of time on task and task load on visual inspection performance in a four-month field study with screeners. At an international airport, 22 screeners inspected X-ray images of cabin baggage for up to 60 min, while a control group (N = 19) screened for 20 min. Hit rate remained stable for low and average task loads. However, when the task load was high, the screeners compensated by speeding up X-ray image inspection at the expense of the hit rate over time on task. Our results support the dynamic-allocation resource theory. Moreover, extending the permitted screening duration to 30 or 40 min should be considered.

Vigilance End-Spurt Patterns in Event-Related Potentials

The end-spurt effect, where performance declines with time-on-task and then increases toward the end of a task, has garnered little attention in the vigilance literature. Researchers have suggested the performance enhancement is due to increased motivation or arousal with knowledge of the end of the vigil. However, recent examination of neural signature patterns during a simultaneous discrimination task, where task length was unknown, provided preliminary support that the end-spurt reflects pacing of resources. The current effort expands this previous work to an additional simultaneous task and to a successive discrimination task across two sessions, one where task length was not known and one where task length was known. Twenty-eight (Study 1) and a separate 24 (Study 2) participants completed a Simultaneous Radar task (Study 1) in one session and Simultaneous and Successive Lines tasks (Study 2) across two sessions while neural data was collected. Several event-related potentials exhibited non-monotonic patterns during the vigilance tasks, in some cases reflecting end-spurt patterns, but more commonly reflecting higher-order polynomial patterns. These patterns were more prevalent in anterior regions as opposed to posterior regions. Of note, the N1 anterior exhibited consistent general patterns across all the vigilance tasks and across sessions. Importantly, even when participants had knowledge of session length, some ERPs still exhibited higher-order polynomial trends, suggesting pacing rather than an end-spurt from motivation or arousal as the vigil ends. These insights can help inform predictive modeling of vigilance performance and the implementation of mitigation efforts to allay the vigilance decrement.

Performance, Hemodynamics, and Stress in a Two-Day Vigilance Task: Practical and Theoretical Implications

OBJECTIVE

To explore vigilance task performance, cerebral blood flow velocity (CBFV), workload, and stress in a within-subjects, two-session experiment.

BACKGROUND

Vigilance, or sustained attention, tasks are often characterized by a decline in operator performance and CBFV with time on task, and high workload and stress. Though performance is known to improve with practice, past research has not included measures of CBFV, stress, and workload in a within-subjects multi-session design, which may also provide insight into ongoing theoretical debate.

METHOD

Participants performed a vigilance task on two separate occasions. Performance, CBFV, workload, and self-reported stress were measured.

RESULTS

Within each session, results were consistent with the vigilance profile found in prior research. Across sessions, performance improved but the time on task decrement remained. Mean CBFV and workload ratings did not differ between sessions, but participants reported significantly less distress, worry, and engagement after session two compared to one.

CONCLUSION

Though practice may not disrupt the standard vigilance profile, it may serve to improve overall performance and reduce stress. However, repeated exposure may have negative implications for engagement and mind-wandering.

APPLICATION

It is important to better understand the relationship between experience, performance, physiological response, and self-reported stress and workload in vigilance because real-world environments often require operators to do the same task over many occasions. While performance improvement and reduced distress is an encouraging result, the decline in engagement requires further research. Results across sessions fail to provide support to the mind-wandering theory of vigilance.

Evaluation of a New Lightweight EEG Technology for Translational Applications of Passive Brain-Computer Interfaces

Technologies like passive brain-computer interfaces (BCI) can enhance human-machine interaction. Anyhow, there are still shortcomings in terms of easiness of use, reliability, and generalizability that prevent passive-BCI from entering real-life situations. The current work aimed to technologically and methodologically design a new gel-free passive-BCI system for out-of-the-lab employment. The choice of the water-based electrodes and the design of a new lightweight headset met the need for easy-to-wear, comfortable, and highly acceptable technology. The proposed system showed high reliability in both laboratory and realistic settings, performing not significantly different from the gold standard based on gel electrodes. In both cases, the proposed system allowed effective discrimination (AUC > 0.9) between low and high levels of workload, vigilance, and stress even for high temporal resolution (<10 s). Finally, the generalizability of the proposed system has been tested through a cross-task calibration. The system calibrated with the data recorded during the laboratory tasks was able to discriminate the targeted human factors during the realistic task reaching AUC values higher than 0.8 at 40 s of temporal resolution in case of vigilance and workload, and 20 s of temporal resolution for the stress monitoring. These results pave the way for ecologic use of the system, where calibration data of the realistic task are difficult to obtain.

Conflict monitoring or multi-tasking? Tracking within-task performance in single-item and multi-item Stroop tasks

Cognitive control is applied in situations that require overriding a habitual and automatic response. The conflict monitoring hypothesis and the Expected Value of Control (EVC) theory as its extension posit a control system responsible for detecting conflicting occasions and adapting to them dynamically within a task. Here we evaluate this prediction in two versions of one of the most popular tasks in cognitive control, namely the Stroop task. We hypothesized that nearby-items interference combines with task interference in the multi-item version effectively turning it into a multi-task that may challenge cognitive control. Adopting an alternative methodology tracking within-task performance, we compared the classical multi-item version of the Stroop task and its single-item counterpart in adults and children. The results revealed a within-task performance decline only in the multi-item version of the task, in both incongruent and neutral conditions, modulated by the presumed maturity of the control system. These findings suggest capacity constraints in control implementation and allocation under conditions requiring parallel execution of multiple cognitive tasks. Task complexity and demands seem to modulate effects on performance. We discuss implications for cognitive control as well as substantial concerns regarding the calculation and use of indices of interference based on the commonly used multi-item version of the Stroop task.

Cognitive load mitigates the executive but not the arousal vigilance decrement

Previous research has shown opposite effects of dual tasking on the vigilance decrement phenomenon. We examined the executive (i.e., detecting infrequent critical signals) and arousal (i.e., sustaining a fast reaction to stimuli without much control on responses) vigilance decrements as a function of task load. Ninety-six participants performed either a single signal-detection (i.e., executive vigilance) task, a single reaction time (i.e., arousal vigilance) task, or a dual vigilance task with the same stimuli and procedure. All participants self-reported their fatigue' state along the session. Exploratory analyses included data from a previous study with a triple task condition. Task load significantly modulated the executive but not the arousal vigilance decrement. Interestingly, the largest increase in mental fatigue was observed in the single executive vigilance task condition. We discuss limitations of classic vigilance theories to account for the vigilance decrement and changes in mental fatigue as a function of task load.

'I need a break': the effect of choice of rest break duration on vigilance

Vigilance is the ability to sustain attention for an extended period of time and to respond to infrequently occurring critical signals. One of the most replicable findings within the vigilance literature is the performance decrement; the decline in performance as time on task increases. In an effort to attenuate the decrement, and decrease the workload and stress associated with vigilance, the present study investigated the role of choice of rest break duration on vigilance performance, perceived workload, and stress. Participants were assigned to one of three conditions: (1) choice condition, (2) no-choice condition (yoked-control), and (3) a no-break control condition. Participants completed a sensory vigilance task and common measures of workload and stress. A vigilance decrement was observed in all conditions. Participants in the choice condition exhibited more conservative responses and fewer false alarms than the no-choice condition. Across all conditions, task engagement and worry decreased, and distress increased. Practitioner Summary: This study shows the impact of rest breaks and autonomy on vigilance task performance. The findings suggest that resource theory is a plausible explanation for the vigilance decrement. Additionally, providing a choice in rest break length changes the operator's criterion following the break. Abbreviations: TSA: transportation security administration; SART: sustained attention to response task; ERP: event-related potential; S-DT: self-determination theory; ISI: interstimulus interval; DSSQ: dundee stress state questionnaire; CFQ: cognitive failures questionnaire; BP: boredom proneness; NASA-TLX: NASA task load index; IMI: intrinsic motivation inventory.

Mind Wandering Influences EEG Signal in Complex Multimodal Environments

The phenomenon of mind wandering (MW), as a family of experiences related to internally directed cognition, heavily influences vigilance evolution. In particular, humans in teleoperations monitoring partially automated fleet before assuming manual control whenever necessary may see their attention drift due to internal sources; as such, it could play an important role in the emergence of out-of-the-loop (OOTL) situations and associated performance problems. To follow, quantify, and mitigate this phenomenon, electroencephalogram (EEG) systems already demonstrated robust results. As MW creates an attentional decoupling, both ERPs and brain oscillations are impacted. However, the factors influencing these markers in complex environments are still not fully understood. In this paper, we specifically addressed the possibility of gradual emergence of attentional decoupling and the differences created by the sensory modality used to convey targets. Eighteen participants were asked to (1) supervise an automated drone performing an obstacle avoidance task (visual task) and (2) respond to infrequent beeps as fast as possible (auditory task). We measured event-related potentials and alpha waves through EEG. We also added a 40-Hz amplitude modulated brown noise to evoke steady-state auditory response (ASSR). Reported MW episodes were categorized between task-related and task-unrelated episodes. We found that N1 ERP component elicited by beeps had lower amplitude during task-unrelated MW, whereas P3 component had higher amplitude during task-related MW, compared with other attentional states. Focusing on parieto-occipital regions, alpha-wave activity was higher during task-unrelated MW compared with others. These results support the decoupling hypothesis for task-unrelated MW but not task-related MW, highlighting possible variations in the "depth" of decoupling depending on MW episodes. Finally, we found no influence of attentional states on ASSR amplitude. We discuss possible reasons explaining why. Results underline both the ability of EEG to track and study MW in laboratory tasks mimicking ecological environments, as well as the complex influence of perceptual decoupling on operators' behavior and, in particular, EEG measures.