Effects of Reliable and Unreliable Automation on Subjective Measures of Mental Workload, Situation Awareness, Trust and Confidence in a Dynamic Flight Task

Eight military pilots completed a study of human-interaction with reliable and unreliable automation in a complex air-to-ground search and destroy mission. Each mission consisted of four distinct stages. Automation was available in three of the four stages to aid the pilot in completing the required tasks. Automation reliability level was manipulated and combined factorially with two levels of workload, resulting in sixteen mission scenarios. Subjective measures of mental workload, situation awareness, trust, confidence, and judgments of the automation reliability were recorded after each mission. An analysis of the subjective measure ratings by mission workload level and the reliability of the first stage automation was performed. The results indicated that pilot trust in the automation was higher when the workload level was low and that there was a significant difference in confidence ratings between high and low workload levels, but only when the automation was unreliable. Design implications are discussed.

Selecting Methods for the Analysis of Reliance on Automation

Human reliance on imperfect automation has been the subject of many laboratory experiments. Across these studies, a diverse set of indices of reliance have been used, even in studies with similar experiment settings. This inconsistency of reliance measures makes the meta-analysis of experimental findings difficult. Moreover, few researchers have rigorously defined the optimal level of automation reliance in their settings, making it difficult to judge the appropriateness of that reliance. This paper attempts to guide researchers in selecting and interpreting measures of automation reliance behavior. We review the reliance analysis methods in existing research and propose four criteria for selecting among them. It is recommended that, where possible, future studies define optimal reliance to make unequivocal judgment about the appropriateness of reliance. In addition, more reliable and insightful conclusions can be obtained through the use of multiple measures.

Effects of conflict alerting system reliability and task difficulty on pilots' conflict detection with cockpit display of traffic information

A total of 24 pilots viewed dynamic encounters between their own aircraft and an intruder aircraft on a 2-D cockpit display of traffic information (CDTI) and estimated the point and time of closest approach. A three-level alerting system provided a correct categorical estimate of the projected miss distance on 83% of the trials. The remaining 17% of alerts were equally divided between misses and false alarms, of large and small magnitude. Roughly half the pilots depended on automation to improve estimation of miss distance relative to the baseline pilots, who viewed identical trials without the aid of automated alerts. Moreover, they did so more on the more difficult traffic trials resulting in improved performance on the 83% correct automation trials without causing harm on the 17% automation-error trials, compared to the baseline group. The automated alerts appeared to lead pilots to inspect the raw data more closely. While assisting the accurate prediction of miss distance, the automation led to an underestimate of the time remaining until the point of closest approach. The results point to the benefits of even imperfect automation in the strategic alerts characteristic of the CDTI, at least as long as this reliability remains high (above 80%).