Using Situation Awareness and Workload to Predict Performance in Submarine Track Management: A Multilevel Approach

Return-to-Manual Performance can be Predicted Before Automation Fails

OBJECTIVE

This study aimed to examine operator state variables (workload, fatigue, and trust in automation) that may predict return-to-manual (RTM) performance when automation fails in simulated air traffic control.

BACKGROUND

Prior research has largely focused on triggering adaptive automation based on reactive indicators of performance degradation or operator strain. A more direct and effective approach may be to proactively engage/disengage automation based on predicted operator RTM performance (conflict detection accuracy and response time), which requires analyses of within-person effects.

METHOD

Participants accepted and handed-off aircraft from their sector and were assisted by imperfect conflict detection/resolution automation. To avoid aircraft conflicts, participants were required to intervene when automation failed to detect a conflict. Participants periodically rated their workload, fatigue and trust in automation.

RESULTS

For participants with the same or higher average trust than the sample average, an increase in their trust (relative to their own average) slowed their subsequent RTM response time. For participants with lower average fatigue than the sample average, an increase in their fatigue (relative to own average) improved their subsequent RTM response time. There was no effect of workload on RTM performance.

CONCLUSIONS

RTM performance degraded as trust in automation increased relative to participants' own average, but only for individuals with average or high levels of trust.

APPLICATIONS

Study outcomes indicate a potential for future adaptive automation systems to detect vulnerable operator states in order to predict subsequent RTM performance decrements.

Sharing imagery and analysis tools in a simulated submarine control room

We examined the impact of sharing periscope imagery and analysis tools in eighteen five-member teams (Sonar, Periscope, 2xTrack Motion Analysts, Track Manager) who undertook simulated submarine patrol tasks. Compared to a baseline condition, sharing imagery to team members increased perceived individual workload, with no improvement to team performance (tactical picture accuracy). When both imagery and analysis tools were shared, perceived workload increased and tactical picture compilation was more accurate. Despite this improved tactical picture for the imagery and analysis tools condition, there was no advantage to mission completion (rendezvous/close contact detection) or situation awareness. In contrast to the increased subjective workload, individuals in teams provided with shared imagery (with or without tools) had a lower physiological response (heart rate, electrodermal) to task demands compared to the baseline condition. Sharing imagery and analysis tools likely benefited tactical picture compilation by enabling dynamic task redistribution and multiple streams of concurrent data analysis.

Optimizing aid activation in adaptive and non-adaptive aiding systems: A framework for design and validation

Development of adaptive aids to support human performance in complex systems is a cornerstone of human factors. Research in this area has led to a diversity of ideas regarding potential activation methods. However, little guidance has been provided on how to select among aid activation methods, and this lack of guidance could hinder adaptive aid development and deployment. Within the current paper, we review available methods of aid activation and describe a process for developing and validating adaptive aiding systems. We focus on supporting system designers who wish to select the ideal aid activation method for an intended application. The process that we recommend is an empirical approach to evaluate the feasibility, costs, and benefits of various potential methods of aid activation. This methodological framework will support practitioners making critical decisions about the design of aiding systems.

Using Past and Present Indicators of Human Workload to Explain Variance in Human Performance

Cognitive workload is assumed to influence performance due to resource competition. However, there is a lack of evidence for a direct relationship between changes in workload within an individual over time and changes in that individual's performance. We collected performance data using a multiple object-tracking task in which we measured workload objectively in real-time using a modified detection response task. Using a multi-level Bayesian model controlling for task difficulty and past performance, we found strong evidence that workload both during and preceding a tracking trial was predictive of performance, such that higher workload led to poorer performance. These negative workload-performance relationships were remarkably consistent across individuals. Importantly, we demonstrate that fluctuations in workload independent from the task demands accounted for significant performance variation. The outcomes have implications for designing real-time adaptive systems to proactively mitigate human performance decrements, but also highlight the pervasive influence of cognitive workload more generally.

A Systematic Review and Meta-Analysis of Direct Objective Measures of Situation Awareness: A Comparison of SAGAT and SPAM

OBJECTIVE

To examine evidence of sensitivity, predictiveness, and methodological concerns regarding direct, objective measures of situation awareness (SA).

BACKGROUND

The ability to objectively measure SA is important to the evaluation of user interfaces and displays, training programs, and automation initiatives, as well as for studies that seek to better understand SA in both individuals and teams. A number of methodological criticisms have been raised creating significant confusion in the research field.

METHOD

A meta-analysis of 243 studies was conducted to examine evidence of sensitivity and predictiveness, and to address methodological questions regarding Situation Awareness Global Assessment Technique (SAGAT), Situation Present Assessment Technique (SPAM), and their variants.

RESULTS

SAGAT and SPAM were found to be equally predictive of performance. SPAM (64%) and real-time probes (73%) were found to have significantly lower sensitivity in comparison to SAGAT (94%). While SAGAT was found not to be overly memory reliant nor intrusive into operator performance, SPAM resulted in problems with intrusiveness in 40% of the studies examined, as well as problems with speed-accuracy tradeoffs, sampling bias, and confounds with workload. Concerns about memory reliance, the utility of these measures for assessing Team SA, and other issues are also addressed.

CONCLUSION

SAGAT was found to be a highly sensitive, reliable, and predictive measure of SA that is useful across a wide variety of domains and experimental settings.

APPLICATION

Direct, objective SA measurement provides useful and diagnostic insights for research and design in a wide variety of domains and study objectives.

Individual differences in higher-level cognitive abilities do not predict overconfidence in complex task performance

Even when people perform tasks poorly, they often report unrealistically positive estimates of their own abilities in these situations. To better understand the origins of such overconfidence, we investigated whether it could be predicted by individual differences in working memory, attentional control, and self-reported trait impulsivity. Overconfidence was estimated by contrasting objective and subjective measures of situation awareness (the ability to perceive and understand task-relevant information in the environment), acquired during a challenging air traffic control simulation. We found no significant relationships between overconfidence and either working memory or attentional control. However, increased impulsivity significantly predicted greater overconfidence. In addition, overall levels of overconfidence were lower in our complex task than in previous studies that used less-complex lab-based tasks. Our results suggest that overconfidence may not be linked to high-level cognitive abilities, but that dynamic tasks with frequent opportunities for performance feedback may reduce misconceptions about personal performance.