Transitioning to future air traffic management: effects of imperfect automation on controller attention and performance

The relationship of trust and dependence

The concepts of automation trust and dependence have often been viewed as closely related and on occasion, have been conflated in the research community. Yet, trust is a cognitive attitude and dependence is a behavioural measure, so it is unsurprising that different factors can affect the two. Here, we review the literature on the correlation between trust and dependence. On average, this correlation across people was quite low, suggesting that people who are more trusting of automation do not necessarily depend upon it more. Separately, we examined experiments that explicitly manipulated the reliability of automation, finding that higher automation reliability increased trust ratings twice as fast as dependence behaviours. This review provides novel quantitative evidence that the two constructs are not strongly correlated. Implications of this work, including potential moderating variables, contexts where trust is still relevant, and considerations of trust measurement, are discussed.

The effect of known decision support reliability on outcome quality and visual information foraging in joint decision making

Decision support systems (DSSs) are being woven into human workflows from aviation to medicine. In this study, we examine decision quality and visual information foraging for DSSs with different known reliability levels. Thirty-six participants completed a financial fraud detection task, first unsupported and then supported by a DSS which highlighted important information sources. Participants were randomly allocated to four cohorts, being informed that the system's reliability was 100%, 90%, 80% or undisclosed. Results showed that only a DSS known to be 100% reliable resulted in participants systematically following its suggestions, increasing the percentage of correct classifications to a median of 100% while halving both, decision time and number of visually attended information sources. In all other conditions, the DSS had no effect on most visual sampling metrics, while decision quality of the human-DSS team was below the reliability level of the DSS. Knowledge of an even slightly unreliable system hence had a profound impact on joint decision making, with participants trusting their significantly worse performance more than the DSSs suggestions.

Testing the Efficacy of Human-Human Trust Repair Strategies with Machines

Trust is a critical component to both human-automation and human-human interactions. Interface manipulations, such as visual anthropomorphism and machine politeness, have been used to affect trust in automation. However, these design strategies are meant to primarily facilitate initial trust formation but have not been examined as a means to actively repair trust that has been violated by a system failure. Previous research has shown that trust in another party can be effectively repaired after a violation using various strategies, but there is little evidence substantiating such strategies in human-automation context. The current study will examine the effectiveness of trust repair strategies, derived from a human-human or human-organizational context, in human-automation interaction.

Intuitive Cognition and Models of Human-Automation Interaction

OBJECTIVE

The aim of this study was to provide an analysis of the implications of the dominance of intuitive cognition in human reasoning and decision making for conceptualizing models and taxonomies of human-automation interaction, focusing on the Parasuraman et al. model and taxonomy.

BACKGROUND

Knowledge about how humans reason and make decisions, which has been shown to be largely intuitive, has implications for the design of future human-machine systems.

METHOD

One hundred twenty articles and books cited in other works as well as those obtained from an Internet search were reviewed. Works were deemed eligible if they were published within the past 50 years and common to a given literature.

RESULTS

Analysis shows that intuitive cognition dominates human reasoning and decision making in all situations examined. The implications of the dominance of intuitive cognition for the Parasuraman et al. model and taxonomy are discussed. A taxonomy of human-automation interaction that incorporates intuitive cognition is suggested.

APPLICATION

Understanding the ways in which human reasoning and decision making is intuitive can provide insight for future models and taxonomies of human-automation interaction.

Benefits of Imperfect Conflict Resolution Advisory Aids for Future Air Traffic Control

OBJECTIVE

The aim of this study was to examine the human-automation interaction issues and the interacting factors in the context of conflict detection and resolution advisory (CRA) systems.

BACKGROUND

The issues of imperfect automation in air traffic control (ATC) have been well documented in previous studies, particularly in conflict-alerting systems. The extent to which the prior findings can be applied to an integrated conflict detection and resolution system in future ATC remains unknown.

METHOD

Twenty-four participants were evenly divided into two groups corresponding to a medium- and a high-traffic density condition, respectively. In each traffic density condition, participants were instructed to perform simulated ATC tasks under four automation conditions, including reliable, unreliable with short time allowance to secondary conflict (TAS), unreliable with long TAS, and manual conditions. Dependent variables accounted for conflict resolution performance, workload, situation awareness, and trust in and dependence on the CRA aid, respectively.

RESULTS

Imposing the CRA automation did increase performance and reduce workload as compared with manual performance. The CRA aid did not decrease situation awareness. The benefits of the CRA aid were manifest even when it was imperfectly reliable and were apparent across traffic loads. In the unreliable blocks, trust in the CRA aid was degraded but dependence was not influenced, yet the performance was not adversely affected.

CONCLUSION

The use of CRA aid would benefit ATC operations across traffic densities.

APPLICATION

CRA aid offers benefits across traffic densities, regardless of its imperfection, as long as its reliability level is set above the threshold of assistance, suggesting its application for future ATC.

Workload Capacity: A Response Time-Based Measure of Automation Dependence

OBJECTIVE

An experiment used the workload capacity measure C(t) to quantify the processing efficiency of human-automation teams and identify operators' automation usage strategies in a speeded decision task.

BACKGROUND

Although response accuracy rates and related measures are often used to measure the influence of an automated decision aid on human performance, aids can also influence response speed. Mean response times (RTs), however, conflate the influence of the human operator and the automated aid on team performance and may mask changes in the operator's performance strategy under aided conditions. The present study used a measure of parallel processing efficiency, or workload capacity, derived from empirical RT distributions as a novel gauge of human-automation performance and automation dependence in a speeded task.

METHOD

Participants performed a speeded probabilistic decision task with and without the assistance of an automated aid. RT distributions were used to calculate two variants of a workload capacity measure, COR(t) and CAND(t).

RESULTS

Capacity measures gave evidence that a diagnosis from the automated aid speeded human participants' responses, and that participants did not moderate their own decision times in anticipation of diagnoses from the aid.

CONCLUSION AND APPLICATION

Workload capacity provides a sensitive and informative measure of human-automation performance and operators' automation dependence in speeded tasks.

A Meta-Analysis of Factors Influencing the Development of Trust in Automation: Implications for Understanding Autonomy in Future Systems

OBJECTIVE

We used meta-analysis to assess research concerning human trust in automation to understand the foundation upon which future autonomous systems can be built.

BACKGROUND

Trust is increasingly important in the growing need for synergistic human-machine teaming. Thus, we expand on our previous meta-analytic foundation in the field of human-robot interaction to include all of automation interaction.

METHOD

We used meta-analysis to assess trust in automation. Thirty studies provided 164 pairwise effect sizes, and 16 studies provided 63 correlational effect sizes.

RESULTS

The overall effect size of all factors on trust development was ḡ = +0.48, and the correlational effect was [Formula: see text]  = +0.34, each of which represented medium effects. Moderator effects were observed for the human-related (ḡ  = +0.49; [Formula: see text] = +0.16) and automation-related (ḡ = +0.53; [Formula: see text] = +0.41) factors. Moderator effects specific to environmental factors proved insufficient in number to calculate at this time.

CONCLUSION

Findings provide a quantitative representation of factors influencing the development of trust in automation as well as identify additional areas of needed empirical research.

APPLICATION

This work has important implications to the enhancement of current and future human-automation interaction, especially in high-risk or extreme performance environments.

Effects of extended lay-off periods on performance and operator trust under adaptable automation

Little is known about the long-term effects of system reliability when operators do not use a system during an extended lay-off period. To examine threats to skill maintenance, 28 participants operated twice a simulation of a complex process control system for 2.5 h, with an 8-month retention interval between sessions. Operators were provided with an adaptable support system, which operated at one of the following reliability levels: 60%, 80% or 100%. Results showed that performance, workload, and trust remained stable at the second testing session, but operators lost self-confidence in their system management abilities. Finally, the effects of system reliability observed at the first testing session were largely found again at the second session. The findings overall suggest that adaptable automation may be a promising means to support operators in maintaining their performance at the second testing session.

Trust in automation: integrating empirical evidence on factors that influence trust

OBJECTIVE

We systematically review recent empirical research on factors that influence trust in automation to present a three-layered trust model that synthesizes existing knowledge.

BACKGROUND

Much of the existing research on factors that guide human-automation interaction is centered around trust, a variable that often determines the willingness of human operators to rely on automation. Studies have utilized a variety of different automated systems in diverse experimental paradigms to identify factors that impact operators' trust.

METHOD

We performed a systematic review of empirical research on trust in automation from January 2002 to June 2013. Papers were deemed eligible only if they reported the results of a human-subjects experiment in which humans interacted with an automated system in order to achieve a goal. Additionally, a relationship between trust (or a trust-related behavior) and another variable had to be measured. All together, 101 total papers, containing 127 eligible studies, were included in the review.

RESULTS

Our analysis revealed three layers of variability in human-automation trust (dispositional trust, situational trust, and learned trust), which we organize into a model. We propose design recommendations for creating trustworthy automation and identify environmental conditions that can affect the strength of the relationship between trust and reliance. Future research directions are also discussed for each layer of trust.

CONCLUSION

Our three-layered trust model provides a new lens for conceptualizing the variability of trust in automation. Its structure can be applied to help guide future research and develop training interventions and design procedures that encourage appropriate trust.

Human factors assessment of conflict resolution aid reliability and time pressure in future air traffic control

Though it has been reported that air traffic controllers' (ATCos') performance improves with the aid of a conflict resolution aid (CRA), the effects of imperfect automation on CRA are so far unknown. The main objective of this study was to examine the effects of imperfect automation on conflict resolution. Twelve students with ATC knowledge were instructed to complete ATC tasks in four CRA conditions including reliable, unreliable and high time pressure, unreliable and low time pressure, and manual conditions. Participants were able to resolve the designated conflicts more accurately and faster in the reliable versus unreliable CRA conditions. When comparing the unreliable CRA and manual conditions, unreliable CRA led to better conflict resolution performance and higher situation awareness. Surprisingly, high time pressure triggered better conflict resolution performance as compared to the low time pressure condition. The findings from the present study highlight the importance of CRA in future ATC operations. Practitioner Summary: Conflict resolution aid (CRA) is a proposed automation decision aid in air traffic control (ATC). It was found in the present study that CRA was able to promote air traffic controllers' performance even when it was not perfectly reliable. These findings highlight the importance of CRA in future ATC operations.

On-the-fly scheduling as a manifestation of partial-order planning and dynamic task values

OBJECTIVE

The aim of this study was to develop a computational account of the spontaneous task ordering that occurs within jobs as work unfolds ("on-the-fly task scheduling").

BACKGROUND

Air traffic control is an example of work in which operators have to schedule their tasks as a partially predictable work flow emerges. To date, little attention has been paid to such on-the-fly scheduling situations.

METHOD

We present a series of discrete-event models fit to conflict resolution decision data collected from experienced controllers operating in a high-fidelity simulation.

RESULTS

Our simulations reveal air traffic controllers' scheduling decisions as examples of the partial-order planning approach of Hayes-Roth and Hayes-Roth. The most successful model uses opportunistic first-come-first-served scheduling to select tasks from a queue. Tasks with short deadlines are executed immediately. Tasks with long deadlines are evaluated to assess whether they need to be executed immediately or deferred.

CONCLUSION

On-the-fly task scheduling is computationally tractable despite its surface complexity and understandable as an example of both the partial-order planning strategy and the dynamic-value approach to prioritization.

Understanding reliance on automation: effects of error type, error distribution, age and experience

An obstacle detection task supported by "imperfect" automation was used with the goal of understanding the effects of automation error types and age on automation reliance. Sixty younger and sixty older adults interacted with a multi-task simulation of an agricultural vehicle (i.e. a virtual harvesting combine). The simulator included an obstacle detection task and a fully manual tracking task. A micro-level analysis provided insight into the way reliance patterns change over time. The results indicated that there are distinct patterns of reliance that develop as a function of error type. A prevalence of automation false alarms led participants to under-rely on the automation during alarm states while over relying on it during non-alarms states. Conversely, a prevalence of automation misses led participants to over-rely on automated alarms and under-rely on the automation during non-alarm states. Older adults adjusted their behavior according to the characteristics of the automation similarly to younger adults, although it took them longer to do so. The results of this study suggest the relationship between automation reliability and reliance depends on the prevalence of specific errors and on the state of the system. Understanding the effects of automation detection criterion settings on human-automation interaction can help designers of automated systems make predictions about human behavior and system performance as a function of the characteristics of the automation.

Relative position vectors: an alternative approach to conflict detection in air traffic control

OBJECTIVE

We explore whether the visual presentation of relative position vectors (RPVs) improves conflict detection in conditions representing some aspects of future airspace concepts.

BACKGROUND

To help air traffic controllers manage increasing traffic, new tools and systems can automate more cognitively demanding processes, such as conflict detection. However, some studies reveal adverse effects of such tools, such as reduced situation awareness and increased workload. New displays are needed that help air traffic controllers handle increasing traffic loads.

METHOD

A new display tool based on the display of RPVs, the Multi-Conflict Display (MCD), is evaluated in a series of simulated conflict detection tasks. The conflict detection performance of air traffic controllers with the MCD plus a conventional plan-view radar display is compared with their performance with a conventional plan-view radar display alone.

RESULTS

Performance with the MCD plus radar was better than with radar alone in complex scenarios requiring controllers to find all actual or potential conflicts, especially when the number of aircraft on the screen was large. However performance with radar alone was better for static scenarios in which conflicts for a target aircraft, or target pair of aircraft, were the focus.

CONCLUSION

Complementing the conventional plan-view display with an RPV display may help controllers detect conflicts more accurately with extremely high aircraft counts.

APPLICATIONS

We provide an initial proof of concept that RPVs may be useful for supporting conflict detection in situations that are partially representative of conditions in which controllers will be working in the future.

Imperfect Automation in Scheduling Operator Attention on Control of Multi-Robots

An operator’s workload increases substantially when the operator must control multiple robots and continually shift attention from robot to robot. As the number of robots increases, the amount of time an operator can spend operating any particular robot decreases, which leads to inevitable changes in the robot’s performance. If the robots could self-report encountered faults, the operator could conserve cognitive resources to spend on reasoning about more complex situations. In the reported experiment, participants performed foraging tasks while assisted by an alarmed system, either Open-queue in which all alarms are displayed or SJF-queue (shortest-job-first), whose reliability level was high (90%) or low (50%) under different task load (3-robots vs. 6-robots). The results showed that simply increasing the system reliability might not effectively contribute to the overall performance or the participants’ trust in automation. An inverse relationship was observed between experienced workload and rated trust which also amplified the effects of imperfect automation.