Measuring the Efficiency of Automation-Aided Performance in a Simulated Baggage Screening Task

Benchmarking automation-aided performance in a forensic face matching task

Carragher and Hancock (2023) investigated how individuals performed in a one-to-one face matching task when assisted by an Automated Facial Recognition System (AFRS). Across five pre-registered experiments they found evidence of suboptimal aided performance, with AFRS-assisted individuals consistently failing to reach the level of performance the AFRS achieved alone. The current study reanalyses these data (Carragher and Hancock, 2023), to benchmark automation-aided performance against a series of statistical models of collaborative decision making, spanning a range of efficiency levels. Analyses using a Bayesian hierarchical signal detection model revealed that collaborative performance was highly inefficient, falling closest to the most suboptimal models of automation dependence tested. This pattern of results generalises previous reports of suboptimal human-automation interaction across a range of visual search, target detection, sensory discrimination, and numeric estimation decision-making tasks. The current study is the first to provide benchmarks of automation-aided performance in the one-to-one face matching task.

Selective Access to Decision Support as a Function of Event Uncertainty

OBJECTIVE

We investigate the impact of event uncertainty, decision support (DS) display format, and DS sensitivity on participants' behavior, performance, subjective workload, and perception of DS usefulness and performance in a classification task.

BACKGROUND

DS systems can positively and negatively affect decision accuracy, performance time, and workload. The ability to access DS selectively, based on informational needs, might improve DS effectiveness.

METHOD

Participants performed a sensory classification task in which they were allowed to request DS on a trial-by-trial basis. DS was presented in separated-binary (SB), separated-likelihood (SL), or integrated-likelihood (IL) formats. Access preferences, task performance, performance time, subjective workload, and perceived DS usefulness and performance were recorded.

RESULTS

Participants accessed DS more often when it was highly sensitive, stimulus information was highly uncertain, or the DS cue and stimulus information were perceptually integrated. Effective sensitivity was highest with the integrated likelihood DS. Although the separated likelihood DS provided more information than the binary likelihood DS, it was accessed less often, leading to lower sensitivity.

CONCLUSION

Participants are most likely to access DS when raw stimulus information is highly uncertain and appear to make effective use of likelihood DS only when DS cues are integrated with raw stimulus information within a display.

APPLICATION

Results suggest that DS use will be most effective when likelihood DS cues and raw stimulus information are perceptually integrated. When DS cues and raw stimuli cannot be perceptually integrated, binary cues from the DS will be more effective than likelihood cues.

Trust in automation and the accuracy of human-algorithm teams performing one-to-one face matching tasks

The human face is commonly used for identity verification. While this task was once exclusively performed by humans, technological advancements have seen automated facial recognition systems (AFRS) integrated into many identification scenarios. Although many state-of-the-art AFRS are exceptionally accurate, they often require human oversight or involvement, such that a human operator actions the final decision. Previously, we have shown that on average, humans assisted by a simulated AFRS (sAFRS) failed to reach the level of accuracy achieved by the same sAFRS alone, due to overturning the system's correct decisions and/or failing to correct sAFRS errors. The aim of the current study was to investigate whether participants' trust in automation was related to their performance on a one-to-one face matching task when assisted by a sAFRS. Participants (n = 160) completed a standard face matching task in two phases: an unassisted baseline phase, and an assisted phase where they were shown the identification decision (95% accurate) made by a sAFRS prior to submitting their own decision. While most participants improved with sAFRS assistance, those with greater relative trust in automation achieved larger gains in performance. However, the average aided performance of participants still failed to reach that of the sAFRS alone, regardless of trust status. Nonetheless, further analysis revealed a small sample of participants who achieved 100% accuracy when aided by the sAFRS. Our results speak to the importance of considering individual differences when selecting employees for roles requiring human-algorithm interaction, including identity verification tasks that incorporate facial recognition technologies.

Interruption in visual search: a systematic review

Visual search, the process of trying to find a target presented among distractors, is a much-studied cognitive task. Less well-studied is the condition in which the search task is interrupted before the target is found. The consequences of such interruptions in visual search have been investigated across various disciplines, which has resulted in diverse and at times contradictory findings. The aim of this systematic review is to provide a more cohesive understanding of the effects of interruptions in visual search. For this purpose, we identified 28 studies that met our inclusion criteria. To facilitate a more organized and comprehensive analysis, we grouped the studies based on three dimensions: the search environment, the interruption aftermath, and the type of the interrupting event. While interruptions in visual search are variable and manifest differently across studies, our review provides a foundational scheme for a more cohesive understanding of the subject. This categorization serves as a starting point for exploring potential future directions, which we delineate in our conclusions.

Beyond the Wizard of Oz: Negative Effects of Imperfect Machine Learning to Examine the Impact of Reliability of Augmented Reality Cues on Visual Search Performance

Despite knowing exactly what an object looks like, searching for it in a person's visual field is a time-consuming and error-prone experience. In Augmented Reality systems, new algorithms are proposed to speed up search time and reduce human errors. However, these algorithms might not always provide 100% accurate visual cues, which might affect users' perceived reliability of the algorithm and, thus, search performance. Here, we examined the detrimental effects of automation bias caused by imperfect cues presented in the Augmented Reality head-mounted display using the YOLOv5 machine learning model. 53 participants in the two groups received either 100% accurate visual cues or 88.9% accurate visual cues. Their performance was compared with the control condition, which did not include any additional cues. The results show how cueing may increase performance and shorten search times. The results also showed that performance with imperfect automation was much worse than perfect automation and that, consistent with automation bias, participants were frequently enticed by incorrect cues.

Knowing When to Pass: The Effect of AI Reliability in Risky Decision Contexts

OBJECTIVE

This study manipulates the presence and reliability of AI recommendations for risky decisions to measure the effect on task performance, behavioral consequences of trust, and deviation from a probability matching collaborative decision-making model.

BACKGROUND

Although AI decision support improves performance, people tend to underutilize AI recommendations, particularly when outcomes are uncertain. As AI reliability increases, task performance improves, largely due to higher rates of compliance (following action recommendations) and reliance (following no-action recommendations).

METHODS

In a between-subject design, participants were assigned to a high reliability AI, low reliability AI, or a control condition. Participants decided whether to bet that their team would win in a series of basketball games tying compensation to performance. We evaluated task performance (in accuracy and signal detection terms) and the behavioral consequences of trust (via compliance and reliance).

RESULTS

AI recommendations improved task performance, had limited impact on risk-taking behavior, and were under-valued by participants. Accuracy, sensitivity (d'), and reliance increased in the high reliability AI condition, but there was no effect on response bias (c) or compliance. Participant behavior was only consistent with a probability matching model for compliance in the low reliability condition.

CONCLUSION

In a pay-off structure that incentivized risk-taking, the primary value of the AI recommendations was in determining when to perform no action (i.e., pass on bets).

APPLICATION

In risky contexts, designers need to consider whether action or no-action recommendations will be more influential to design appropriate interventions.

Supporting detection of hostile intentions: automated assistance in a dynamic decision-making context

In a dynamic decision-making task simulating basic ship movements, participants attempted, through a series of actions, to elicit and identify which one of six other ships was exhibiting either of two hostile behaviors. A high-performing, although imperfect, automated attention aid was introduced. It visually highlighted the ship categorized by an algorithm as the most likely to be hostile. Half of participants also received automation transparency in the form of a statement about why the hostile ship was highlighted. Results indicated that while the aid's advice was often complied with and hence led to higher accuracy with a shorter response time, detection was still suboptimal. Additionally, transparency had limited impacts on all aspects of performance. Implications for detection of hostile intentions and the challenges of supporting dynamic decision making are discussed.

Simple manipulations of anthropomorphism fail to induce perceptions of humanness or improve trust in an automated agent

Although automation is employed as an aid to human performance, operators often interact with automated decision aids inefficiently. The current study investigated whether anthropomorphic automation would engender higher trust and use, subsequently improving human-automation team performance. Participants performed a multi-element probabilistic signal detection task in which they diagnosed a hypothetical nuclear reactor as in a state of safety or danger. The task was completed unassisted and assisted by a 93%-reliable agent varying in anthropomorphism. Results gave no evidence that participants' perceptions of anthropomorphism differed between conditions. Further, anthropomorphic automation failed to bolster trust and automation-aided performance. Findings suggest that the benefits of anthropomorphism may be limited in some contexts.

Challenging presumed technological superiority when working with (artificial) colleagues

Technological advancements are ubiquitously supporting or even replacing humans in all areas of life, bringing the potential for human-technology symbiosis but also novel challenges. To address these challenges, we conducted three experiments in different task contexts ranging from loan assignment over X-Ray evaluation to process industry. Specifically, we investigated the impact of support agent (artificial intelligence, decision support system, or human) and failure experience (one vs. none) on trust-related aspects of human-agent interaction. This included not only the subjective evaluation of the respective agent in terms of trust, reliability, and responsibility, when working together, but also a change in perspective to the willingness to be assessed oneself by the agent. In contrast to a presumed technological superiority, we show a general advantage with regard to trust and responsibility of human support over both technical support systems (i.e., artificial intelligence and decision support system), regardless of task context from the collaborative perspective. This effect reversed to a preference for technical systems when switching the perspective to being assessed. These findings illustrate an imperfect automation schema from the perspective of the advice-taker and demonstrate the importance of perspective when working with or being assessed by machine intelligence.