Intelligent Agent Transparency in Human-Agent Teaming for Multi-UxV Management

How Terminology Affects Users' Responses to System Failures

OBJECTIVE

The objective of our research is to advance the understanding of behavioral responses to a system's error. By examining trust as a dynamic variable and drawing from attribution theory, we explain the underlying mechanism and suggest how terminology can be used to mitigate the so-called algorithm aversion. In this way, we show that the use of different terms may shape consumers' perceptions and provide guidance on how these differences can be mitigated.

BACKGROUND

Previous research has interchangeably used various terms to refer to a system and results regarding trust in systems have been ambiguous.

METHODS

Across three studies, we examine the effect of different system terminology on consumer behavior following a system failure.

RESULTS

Our results show that terminology crucially affects user behavior. Describing a system as "AI" (i.e., self-learning and perceived as more complex) instead of as "algorithmic" (i.e., a less complex rule-based system) leads to more favorable behavioral responses by users when a system error occurs.

CONCLUSION

We suggest that in cases when a system's characteristics do not allow for it to be called "AI," users should be provided with an explanation of why the system's error occurred, and task complexity should be pointed out. We highlight the importance of terminology, as this can unintentionally impact the robustness and replicability of research findings.

APPLICATION

This research offers insights for industries utilizing AI and algorithmic systems, highlighting how strategic terminology use can shape user trust and response to errors, thereby enhancing system acceptance.

Transparent Automated Advice to Mitigate the Impact of Variation in Automation Reliability

OBJECTIVE

To examine the extent to which increased automation transparency can mitigate the potential negative effects of low and high automation reliability on disuse and misuse of automated advice, and perceived trust in automation.

BACKGROUND

Automated decision aids that vary in the reliability of their advice are increasingly used in workplaces. Low-reliability automation can increase disuse of automated advice, while high-reliability automation can increase misuse. These effects could be reduced if the rationale underlying automated advice is made more transparent.

METHODS

Participants selected the optimal UV to complete missions. The Recommender (automated decision aid) assisted participants by providing advice; however, it was not always reliable. Participants determined whether the Recommender provided accurate information and whether to accept or reject advice. The level of automation transparency (medium, high) and reliability (low: 65%, high: 90%) were manipulated between-subjects.

RESULTS

With high- compared to low-reliability automation, participants made more accurate (correctly accepted advice and identified whether information was accurate/inaccurate) and faster decisions, and reported increased trust in automation. Increased transparency led to more accurate and faster decisions, lower subjective workload, and higher usability ratings. It also eliminated the increased automation disuse associated with low-reliability automation. However, transparency did not mitigate the misuse associated with high-reliability automation.

CONCLUSION

Transparency protected against low-reliability automation disuse, but not against the increased misuse potentially associated with the reduced monitoring and verification of high-reliability automation.

APPLICATION

These outcomes can inform the design of transparent automation to improve human-automation teaming under conditions of varied automation reliability.

Enhancing human-AI collaboration: The case of colonoscopy

Diagnostic errors impact patient health and healthcare costs. Artificial Intelligence (AI) shows promise in mitigating this burden by supporting Medical Doctors in decision-making. However, the mere display of excellent or even superhuman performance by AI in specific tasks does not guarantee a positive impact on medical practice. Effective AI assistance should target the primary causes of human errors and foster effective collaborative decision-making with human experts who remain the ultimate decision-makers. In this narrative review, we apply these principles to the specific scenario of AI assistance during colonoscopy. By unraveling the neurocognitive foundations of the colonoscopy procedure, we identify multiple bottlenecks in perception, attention, and decision-making that contribute to diagnostic errors, shedding light on potential interventions to mitigate them. Furthermore, we explored how existing AI devices fare in clinical practice and whether they achieved an optimal integration with the human decision-maker. We argue that to foster optimal Human-AI collaboration, future research should expand our knowledge of factors influencing AI's impact, establish evidence-based cognitive models, and develop training programs based on them. These efforts will enhance human-AI collaboration, ultimately improving diagnostic accuracy and patient outcomes. The principles illuminated in this review hold more general value, extending their relevance to a wide array of medical procedures and beyond.

Agent Transparency, Situation Awareness, Mental Workload, and Operator Performance: A Systematic Literature Review

OBJECTIVE

In this review, we investigate the relationship between agent transparency, Situation Awareness, mental workload, and operator performance for safety critical domains.

BACKGROUND

The advancement of highly sophisticated automation across safety critical domains poses a challenge for effective human oversight. Automation transparency is a design principle that could support humans by making the automation's inner workings observable (i.e., "seeing-into"). However, experimental support for this has not been systematically documented to date.

METHOD

Based on the PRISMA method, a broad and systematic search of the literature was performed focusing on identifying empirical research investigating the effect of transparency on central Human Factors variables.

RESULTS

Our final sample consisted of 17 experimental studies that investigated transparency in a controlled setting. The studies typically employed three human-automation interaction types: responding to agent-generated proposals, supervisory control of agents, and monitoring only. There is an overall trend in the data pointing towards a beneficial effect of transparency. However, the data reveals variations in Situation Awareness, mental workload, and operator performance for specific tasks, agent-types, and level of integration of transparency information in primary task displays.

CONCLUSION

Our data suggests a promising effect of automation transparency on Situation Awareness and operator performance, without the cost of added mental workload, for instances where humans respond to agent-generated proposals and where humans have a supervisory role.

APPLICATION

Strategies to improve human performance when interacting with intelligent agents should focus on allowing humans to see into its information processing stages, considering the integration of information in existing Human Machine Interface solutions.

Can increased automation transparency mitigate the effects of time pressure on automation use?

A factor that can potentially negatively impact the accuracy of automated decision aid use, and increase perceived workload, is time pressure. Increased automation transparency can increase the accuracy of automation use. We examined the extent to which increased transparency can mitigate the negative effects of time pressure on the accuracy of automation use and perceived workload. Participants completed an uninhabited vehicle (UV) management task where they assigned the best UV to complete missions by either accepting or rejecting automated advice. Participants made a decision after either 25s (low time pressure) or 12s (high time pressure). The accuracy of automation use decreased, and perceived workload increased, when under higher time pressure. Higher transparency benefited the accuracy of automation use and increased perceived trust and usability. However, high transparency did not mitigate the negative impacts of high time pressure, indicating that increased time pressure can influence the processing of highly transparent information.

Responsible (use of) AI

Although there is a rich history of philosophical definitions of ethics when applied to human behavior, applying the same concepts and principles to AI may be fraught with problems. Anthropomorphizing AI to have characteristics such as "ethics" may promote a dangerous, unrealistic expectation that AI can be trained to have inherent, guaranteed ethical behavior. The authors instead advocate for increased research into the ethical use of AI from initial ideation and design through operational use and sustainment. The authors advocate for five key research areas: (1) education in ethics and core AI concepts for AI developers, leaders, and users, (2) development and use of model cards or datasheets for datasets to provide transparency into the strengths, limits, and potential biases of a trained model, (3) employing human-centered design that seeks to understand human value structures within a task context and enable effective human-machine interaction through intuitive and transparent interfaces, (4) targeted use of run time assurance that monitors and modifies the inputs or outputs of a trained model when necessary to enforce ethical principles such as safety or limiting bias, and (5) developing best practices for the use of a joint human-AI co-creation and training experience to enable a shared mental model and higher performance through potential emergent behavior.

Supporting detection of hostile intentions: automated assistance in a dynamic decision-making context

In a dynamic decision-making task simulating basic ship movements, participants attempted, through a series of actions, to elicit and identify which one of six other ships was exhibiting either of two hostile behaviors. A high-performing, although imperfect, automated attention aid was introduced. It visually highlighted the ship categorized by an algorithm as the most likely to be hostile. Half of participants also received automation transparency in the form of a statement about why the hostile ship was highlighted. Results indicated that while the aid's advice was often complied with and hence led to higher accuracy with a shorter response time, detection was still suboptimal. Additionally, transparency had limited impacts on all aspects of performance. Implications for detection of hostile intentions and the challenges of supporting dynamic decision making are discussed.

Psychological assessment of AI-based decision support systems: tool development and expected benefits

This study aimed to develop an evaluation tool that assesses the use of AI-based decision support systems (DSSs) in professional practice from a human-centered perspective. Following the International Organization for Standardization, this perspective aims to ensure that the use of interactive technologies improves users' psychological load experience and behavior, e.g., in the form of reduced stress experience or increased performance. Concomitantly, this perspective attempts to proactively prevent or detect and correct the potential negative effects of these technologies on user load, such as impaired satisfaction and engagement, as early as possible. Based on this perspective, we developed and validated a questionnaire instrument, the Psychological Assessment of AI-based DSSs (PAAI), for the user-centered evaluation of the use of AI-based DSSs in practice. In particular, the instrument considers central design characteristics of AI-based DSSs and the corresponding work situation, which have a significant impact on users' psychological load. The instrument was tested in two independent studies. In Study 1, N = 223 individuals were recruited. Based on the results of item and scale analyses and an exploratory factor analysis, the newly developed instrument was refined, and the final version was tested using a confirmatory factor analysis. Findings showed acceptable-to-good fit indices, confirming the factorial validity of the PAAI. This was confirmed in a second study, which had N = 471 participants. Again, the CFA yielded acceptable-to-good fit indices. The validity was further confirmed using convergent and criterion validity analyses.

Do concurrent task demands impact the benefit of automation transparency?

Automated decision aids typically improve decision-making, but incorrect advice risks automation misuse or disuse. We examined the novel question of whether increased automation transparency improves the accuracy of automation use under conditions with/without concurrent (non-automated assisted) task demands. Participants completed an uninhabited vehicle (UV) management task whereby they assigned the best UV to complete missions. Automation advised the best UV but was not always correct. Concurrent non-automated task demands decreased the accuracy of automation use, and increased decision time and perceived workload. With no concurrent task demands, increased transparency which provided more information on how the automation made decisions, improved the accuracy of automation use. With concurrent task demands, increased transparency led to higher trust ratings, faster decisions, and a bias towards agreeing with automation. These outcomes indicate increased reliance on highly transparent automation under conditions with concurrent task demands and have potential implications for human-automation teaming design.

Reliable and transparent in-vehicle agents lead to higher behavioral trust in conditionally automated driving systems

Trust is critical for human-automation collaboration, especially under safety-critical tasks such as driving. Providing explainable information on how the automation system reaches decisions and predictions can improve system transparency, which is believed to further facilitate driver trust and user evaluation of the automated vehicles. However, what the optimal level of transparency is and how the system communicates it to calibrate drivers' trust and improve their driving performance remain uncertain. Such uncertainty becomes even more unpredictable given that the system reliability remains dynamic due to current technological limitations. To address this issue in conditionally automated vehicles, a total of 30 participants were recruited in a driving simulator study and assigned to either a low or a high system reliability condition. They experienced two driving scenarios accompanied by two types of in-vehicle agents delivering information with different transparency types: "what"-then-wait (on-demand) and "what + why" (proactive). The on-demand agent provided some information about the upcoming event and delivered more information if prompted by the driver, whereas the proactive agent provided all information at once. Results indicated that the on-demand agent was more habitable, or naturalistic, to drivers and was perceived with faster system response speed compared to the proactive agent. Drivers under the high-reliability condition complied with the takeover request (TOR) more (if the agent was on-demand) and had shorter takeover times (in both agent conditions) compared to those under the low-reliability condition. These findings inspire how the automation system can deliver information to improve system transparency while adapting to system reliability and user evaluation, which further contributes to driver trust calibration and performance correction in future automated vehicles.

Artificial Intelligence and Groups: Effects of Attitudes and Discretion on Collaboration

We theorize about human-team collaboration with AI by drawing upon research in groups and teams, social psychology, information systems, engineering, and beyond. Based on our review, we focus on two main issues in the teams and AI arena. The first is whether the team generally views AI positively or negatively. The second is whether the decision to use AI is left up to the team members (voluntary use of AI) or mandated by top management or other policy-setters in the organization. These two aspects guide our creation of a team-level conceptual framework modeling how AI introduced as a mandated addition to the team can have asymmetric effects on collaboration level depending on the team’s attitudes about AI. When AI is viewed positively by the team, the effect of mandatory use suppresses collaboration in the team. But when a team has negative attitudes toward AI, mandatory use elevates team collaboration. Our model emphasizes the need for managing team attitudes and discretion strategies and promoting new research directions regarding AI’s implications for teamwork.

A Bayesian Network Approach to Explainable Reinforcement Learning with Distal Information

Nowadays, Artificial Intelligence systems have expanded their competence field from research to industry and daily life, so understanding how they make decisions is becoming fundamental to reducing the lack of trust between users and machines and increasing the transparency of the model. This paper aims to automate the generation of explanations for model-free Reinforcement Learning algorithms by answering "why" and "why not" questions. To this end, we use Bayesian Networks in combination with the NOTEARS algorithm for automatic structure learning. This approach complements an existing framework very well and demonstrates thus a step towards generating explanations with as little user input as possible. This approach is computationally evaluated in three benchmarks using different Reinforcement Learning methods to highlight that it is independent of the type of model used and the explanations are then rated through a human study. The results obtained are compared to other baseline explanation models to underline the satisfying performance of the framework presented in terms of increasing the understanding, transparency and trust in the action chosen by the agent.

Can counterfactual explanations of AI systems' predictions skew lay users' causal intuitions about the world? If so, can we correct for that?

Counterfactual (CF) explanations have been employed as one of the modes of explainability in explainable artificial intelligence (AI)-both to increase the transparency of AI systems and to provide recourse. Cognitive science and psychology have pointed out that people regularly use CFs to express causal relationships. Most AI systems, however, are only able to capture associations or correlations in data, so interpreting them as casual would not be justified. In this perspective, we present two experiments (total n = 364) exploring the effects of CF explanations of AI systems' predictions on lay people's causal beliefs about the real world. In Experiment 1, we found that providing CF explanations of an AI system's predictions does indeed (unjustifiably) affect people's causal beliefs regarding factors/features the AI uses and that people are more likely to view them as causal factors in the real world. Inspired by the literature on misinformation and health warning messaging, Experiment 2 tested whether we can correct for the unjustified change in causal beliefs. We found that pointing out that AI systems capture correlations and not necessarily causal relationships can attenuate the effects of CF explanations on people's causal beliefs.

Explanatory machine learning for justified trust in human-AI collaboration: Experiments on file deletion recommendations

In the digital age, saving and accumulating large amounts of digital data is a common phenomenon. However, saving does not only consume energy, but may also cause information overload and prevent people from staying focused and working effectively. We present and systematically examine an explanatory AI system (Dare2Del), which supports individuals to delete irrelevant digital objects. To give recommendations for the optimization of related human-computer interactions, we vary different design features (explanations, familiarity, verifiability) within and across three experiments (N ₁ = 61, N ₂ = 33, N ₃= 73). Moreover, building on the concept of distributed cognition, we check possible cross-connections between external (digital) and internal (human) memory. Specifically, we examine whether deleting external files also contributes to human forgetting of the related mental representations. Multilevel modeling results show the importance of presenting explanations for the acceptance of deleting suggestions in all three experiments, but also point to the need of their verifiability to generate trust in the system. However, we did not find clear evidence that deleting computer files contributes to human forgetting of the related memories. Based on our findings, we provide basic recommendations for the design of AI systems that can help to reduce the burden on people and the digital environment, and suggest directions for future research.

The Effects of Transparency and Reliability of In-Vehicle Intelligent Agents on Driver Perception, Takeover Performance, Workload and Situation Awareness in Conditionally Automated Vehicles

In the context of automated vehicles, transparency of in-vehicle intelligent agents (IVIAs) is an important contributor to driver perception, situation awareness (SA), and driving performance. However, the effects of agent transparency on driver performance when the agent is unreliable have not been fully examined yet. This paper examined how transparency and reliability of the IVIAs affect drivers’ perception of the agent, takeover performance, workload and SA. A 2 × 2 mixed factorial design was used in this study, with transparency (Push: proactive vs. Pull: on-demand) as a within-subjects variable and reliability (high vs. low) as a between-subjects variable. In a driving simulator, 27 young drivers drove with two types of in-vehicle agents during the conditionally automated driving. Results suggest that transparency influenced participants’ perception on the agent and perceived workload. High reliability agent was associated with higher situation awareness and less effort, compared to low reliability agent. There was an interaction effect between transparency and reliability on takeover performance. These findings could have important implications for the continued design and development of IVIAs of the automated vehicle system.

The influence of interdependence and a transparent or explainable communication style on human-robot teamwork

Humans and robots are increasingly working together in human-robot teams. Teamwork requires communication, especially when interdependence between team members is high. In previous work, we identified a conceptual difference between sharing what you are doing (i.e., being transparent) and why you are doing it (i.e., being explainable). Although the second might sound better, it is important to avoid information overload. Therefore, an online experiment (n = 72) was conducted to study the effect of communication style of a robot (silent, transparent, explainable, or adaptive based on time pressure and relevancy) on human-robot teamwork. We examined the effects of these communication styles on trust in the robot, workload during the task, situation awareness, reliance on the robot, human contribution during the task, human communication frequency, and team performance. Moreover, we included two levels of interdependence between human and robot (high vs. low), since mutual dependency might influence which communication style is best. Participants collaborated with a virtual robot during two simulated search and rescue tasks varying in their level of interdependence. Results confirm that in general robot communication results in more trust in and understanding of the robot, while showing no evidence of a higher workload when the robot communicates or adds explanations to being transparent. Providing explanations, however, did result in more reliance on RescueBot. Furthermore, compared to being silent, only being explainable results a higher situation awareness when interdependence is high. Results further show that being highly interdependent decreases trust, reliance, and team performance while increasing workload and situation awareness. High interdependence also increases human communication if the robot is not silent, human rescue contribution if the robot does not provide explanations, and the strength of the positive association between situation awareness and team performance. From these results, we can conclude that robot communication is crucial for human-robot teamwork, and that important differences exist between being transparent, explainable, or adaptive. Our findings also highlight the fundamental importance of interdependence in studies on explainability in robots.

Human-Autonomy Teaming: A Review and Analysis of the Empirical Literature

OBJECTIVE

We define human-autonomy teaming and offer a synthesis of the existing empirical research on the topic. Specifically, we identify the research environments, dependent variables, themes representing the key findings, and critical future research directions.

BACKGROUND

Whereas a burgeoning literature on high-performance teamwork identifies the factors critical to success, much less is known about how human-autonomy teams (HATs) achieve success. Human-autonomy teamwork involves humans working interdependently toward a common goal along with autonomous agents. Autonomous agents involve a degree of self-government and self-directed behavior (agency), and autonomous agents take on a unique role or set of tasks and work interdependently with human team members to achieve a shared objective.

METHOD

We searched the literature on human-autonomy teaming. To meet our criteria for inclusion, the paper needed to involve empirical research and meet our definition of human-autonomy teaming. We found 76 articles that met our criteria for inclusion.

RESULTS

We report on research environments and we find that the key independent variables involve autonomous agent characteristics, team composition, task characteristics, human individual differences, training, and communication. We identify themes for each of these and discuss the future research needs.

CONCLUSION

There are areas where research findings are clear and consistent, but there are many opportunities for future research. Particularly important will be research that identifies mechanisms linking team input to team output variables.

Research on Transparency Design Based on Shared Situation Awareness in Semi-Automatic Driving

While automated driving has attracted more and more attention, human drivers still play an important role. Drivers must be aware of their surroundings and potential danger to take over the vehicle’s control immediately when the automation system cannot function, which could otherwise pose a road safety risk. The transparent design of the human–machine interface can accurately convey the perceived information of the machine to help drivers understand the dynamic environment. We designed the scenario where the rear vehicle is about to overtake the ego vehicle and has a dangerous tendency to collide with the ego vehicle by stepping on the lane line. We analyzed the information shared by the intelligent system in this scenario to make “transparent” the intelligent system. According to the requirements of the task model of transparency and the three stages of situation awareness, we formed three designs. Then we analyzed the availability, situation awareness, and workload of different designs. Results showed that the transparency design according to the shared situation awareness stage can have a positive impact on the usability of the intelligent system. Therefore, the effective transparency-based design in automated driving can enhance the user’s understanding of the system and improve their experience.

A Methodology for Evaluating Operator Usage of Machine Learning Recommendations for Power Grid Contingency Analysis

This work presents the application of a methodology to measure domain expert trust and workload, elicit feedback, and understand the technological usability and impact when a machine learning assistant is introduced into contingency analysis for real-time power grid simulation. The goal of this framework is to rapidly collect and analyze a broad variety of human factors data in order to accelerate the development and evaluation loop for deploying machine learning applications. We describe our methodology and analysis, and we discuss insights gained from a pilot participant about the current usability state of an early technology readiness level (TRL) artificial neural network (ANN) recommender.

Design of Proactive Interaction for In-Vehicle Robots Based on Transparency

Based on the transparency theory, this study investigates the appropriate amount of transparency information expressed by the in-vehicle robot under two channels of voice and visual in a proactive interaction scenario. The experiments are to test and evaluate different transparency levels and combinations of information in different channels of the in-vehicle robot, based on a driving simulator to collect subjective and objective data, which focuses on users' safety, usability, trust, and emotion dimensions under driving conditions. The results show that appropriate transparency expression is able to improve drivers' driving control and subjective evaluation and that drivers need a different amount of transparency information in different types of tasks.

Exploring the influence of a user-specific explainable virtual advisor on health behaviour change intentions

Virtual advisors (VAs) are being utilised almost in every service nowadays from entertainment to healthcare. To increase the user's trust in these VAs and encourage the users to follow their advice, they should have the capability of explaining their decisions, particularly, when the decision is vital such as health advice. However, the role of an explainable VA in health behaviour change is understudied. There is evidence that people tend to change their intentions towards health behaviour when the persuasion message is linked to their mental state. Thus, this study explores this link by introducing an explainable VA that provides explanation according to the user's mental state (beliefs and goals) rather than the agent's mental state as commonly utilised in explainable agents. It further explores the influence of different explanation patterns that refer to beliefs, goals, or beliefs&goals on the user's behaviour change. An explainable VA was designed to advise undergraduate students how to manage their study-related stress by motivating them to change certain behaviours. With 91 participants, the VA was evaluated and the results revealed that user-specific explanation could significantly encourage behaviour change intentions and build good user-agent relationship. Small differences were found between the three types of explanation patterns.

Future Directions for Human-Centered Transparent Systems for Engine Room Monitoring in Shore Control Centers

Many autonomous ship projects have reflected the increasing interest in incorporating the concept of autonomy into the maritime transportation sector. However, autonomy is not a silver bullet, as exemplified by many incidents in the past involving human and machine interaction; rather it introduces new Human Factor (HF) challenges. These challenges are especially critical for Engine Room Monitoring (ERM) in Shore Control Centre (SCCs) due to the system’s complexity and the absence of human senses in the decision-making process. A transparent system is one of the potential solutions, providing a rationale behind its suggestion. However, diverse implementations of transparency schemes have resulted in prevalent inconsistencies in its effects. This literature review paper investigates 17 transparency studies published over the last eight years to identify (a) different approaches to developing transparent systems, (b) the effects of transparency on key HFs, and (c) the effects of information presentation methods and uncertainty information. The findings suggest that the explicit presentation of information could strengthen the benefits of the transparent system and could be promising for performance improvements in ERM tasks in the SCC.

Contrastive explanation: a structural-model approach

This paper presents a model of contrastive explanation using structural casual models. The topic of causal explanation in artificial intelligence has gathered interest in recent years as researchers and practitioners aim to increase trust and understanding of intelligent decision-making. While different sub-fields of artificial intelligence have looked into this problem with a sub-field-specific view, there are few models that aim to capture explanation more generally. One general model is based on structural causal models. It defines an explanation as a fact that, if found to be true, would constitute an actual cause of a specific event. However, research in philosophy and social sciences shows that explanations are contrastive: that is, when people ask for an explanation of an event—the fact—they (sometimes implicitly) are asking for an explanation relative to some contrast case; that is, ‘Why P rather than Q?’. In this paper, we extend the structural causal model approach to define two complementary notions of contrastive explanation, and demonstrate them on two classical problems in artificial intelligence: classification and planning. We believe that this model can help researchers in subfields of artificial intelligence to better understand contrastive explanation.

Robot Transparency and Anthropomorphic Attribute Effects on Human-Robot Interactions

Anthropomorphic robots need to maintain effective and emotive communication with humans as automotive agents to establish and maintain effective human-robot performances and positive human experiences. Previous research has shown that the characteristics of robot communication positively affect human-robot interaction outcomes such as usability, trust, workload, and performance. In this study, we investigated the characteristics of transparency and anthropomorphism in robotic dual-channel communication, encompassing the voice channel (low or high, increasing the amount of information provided by textual information) and the visual channel (low or high, increasing the amount of information provided by expressive information). The results showed the benefits and limitations of increasing the transparency and anthropomorphism, demonstrating the significance of the careful implementation of transparency methods. The limitations and future directions are discussed.

The Impact of Transparency and Decision Risk on Human-Automation Teaming Outcomes

OBJECTIVE

Examine the effects of decision risk and automation transparency on the accuracy and timeliness of operator decisions, automation verification rates, and subjective workload.

BACKGROUND

Decision aids typically benefit performance, but can provide incorrect advice due to contextual factors, creating the potential for automation disuse or misuse. Decision aids can reduce an operator's manual problem evaluation, and it can also be strategic for operators to minimize verifying automated advice in order to manage workload.

METHOD

Participants assigned the optimal unmanned vehicle to complete missions. A decision aid provided advice but was not always reliable. Two levels of decision aid transparency were manipulated between participants. The risk associated with each decision was manipulated using a financial incentive scheme. Participants could use a calculator to verify automated advice; however, this resulted in a financial penalty.

RESULTS

For high- compared with low-risk decisions, participants were more likely to reject incorrect automated advice and were more likely to verify automation and reported higher workload. Increased transparency did not lead to more accurate decisions and did not impact workload but decreased automation verification and eliminated the increased decision time associated with high decision risk.

CONCLUSION

Increased automation transparency was beneficial in that it decreased automation verification and decreased decision time. The increased workload and automation verification for high-risk missions is not necessarily problematic given the improved automation correct rejection rate.

APPLICATION

The findings have potential application to the design of interfaces to improve human-automation teaming, and for anticipating the impact of decision risk on operator behavior.

Dancing With Algorithms: Interaction Creates Greater Preference and Trust in Machine-Learned Behavior

OBJECTIVE

We examined a method of machine learning (ML) to evaluate its potential to develop more trustworthy control of unmanned vehicle area search behaviors.

BACKGROUND

ML typically lacks interaction with the user. Novel interactive machine learning (IML) techniques incorporate user feedback, enabling observation of emerging ML behaviors, and human collaboration during ML of a task. This may enable trust and recognition of these algorithms.

METHOD

Participants judged and selected behaviors in a low and a high interaction condition (IML) over the course of behavior evolution using ML. User trust in the outputs, as well as preference, and ability to discriminate and recognize the behaviors were measured.

RESULTS

Compared to noninteractive techniques, IML behaviors were more trusted and preferred, as well as recognizable, separate from non-IML behaviors, and approached similar performance as pure ML models.

CONCLUSION

IML shows promise for creating behaviors by involving the user; this is the first extension of this technique for vehicle behavior model development targeting user satisfaction and is unique in its multifaceted evaluation of how users perceived, trusted, and implemented these learned controllers.

APPLICATION

There are many contexts where the brittleness of ML cannot be trusted, but the advantage of ML over traditional programmed behaviors may be large, as in some military operations where they could be scaled. IML in this early form appears to generate satisfactory behaviors without sacrificing performance, use, or trust in the behavior, but more work is necessary.

Inferring Trust From Users' Behaviours; Agents' Predictability Positively Affects Trust, Task Performance and Cognitive Load in Human-Agent Real-Time Collaboration

Collaborative virtual agents help human operators to perform tasks in real-time. For this collaboration to be effective, human operators must appropriately trust the agent(s) they are interacting with. Multiple factors influence trust, such as the context of interaction, prior experiences with automated systems and the quality of the help offered by agents in terms of its transparency and performance. Most of the literature on trust in automation identified the performance of the agent as a key factor influencing trust. However, other work has shown that the behavior of the agent, type of the agent’s errors, and predictability of the agent’s actions can influence the likelihood of the user’s reliance on the agent and efficiency of tasks completion. Our work focuses on how agents’ predictability affects cognitive load, performance and users’ trust in a real-time human-agent collaborative task. We used an interactive aiming task where participants had to collaborate with different agents that varied in terms of their predictability and performance. This setup uses behavioral information (such as task performance and reliance on the agent) as well as standardized survey instruments to estimate participants’ reported trust in the agent, cognitive load and perception of task difficulty. Thirty participants took part in our lab-based study. Our results showed that agents with more predictable behaviors have a more positive impact on task performance, reliance and trust while reducing cognitive workload. In addition, we investigated the human-agent trust relationship by creating models that could predict participants’ trust ratings using interaction data. We found that we could reliably estimate participants’ reported trust in the agents using information related to performance, task difficulty and reliance. This study provides insights on behavioral factors that are the most meaningful to anticipate complacent or distrusting attitudes toward automation. With this work, we seek to pave the way for the development of trust-aware agents capable of responding more appropriately to users by being able to monitor components of the human-agent relationships that are the most salient for trust calibration.

The Perceptual Belief Problem

The explainability of robotic systems depends on people’s ability to reliably attribute perceptual beliefs to robots, i.e., what robots know (or believe) about objects and events in the world based on their perception. However, the perceptual systems of robots are not necessarily well understood by the majority of people interacting with them. In this article, we explain why this is a significant, difficult, and unique problem in social robotics. The inability to judge what a robot knows (and does not know) about the physical environment it shares with people gives rise to a host of communicative and interactive issues, including difficulties to communicate about objects or adapt to events in the environment. The challenge faced by social robotics researchers or designers who want to facilitate appropriate attributions of perceptual beliefs to robots is to shape human–robot interactions so that people understand what robots know about objects and events in the environment. To meet this challenge, we argue, it is necessary to advance our knowledge of when and why people form incorrect or inadequate mental models of robots’ perceptual and cognitive mechanisms. We outline a general approach to studying this empirically and discuss potential solutions to the problem.

Trusting Autonomous Security Robots: The Role of Reliability and Stated Social Intent

OBJECTIVE

This research examined the effects of reliability and stated social intent on trust, trustworthiness, and one's willingness to endorse use of an autonomous security robot (ASR).

BACKGROUND

Human-robot interactions in the domain of security is plausible, yet we know very little about what drives acceptance of ASRs. Past research has used static images and game-based simulations to depict the robots versus actual humans interacting with actual robots.

METHOD

A video depicted an ASR interacting with a human. The ASR reviewed access credentials and allowed entrance once verified. If the ASR could not verify one's credentials it instructed the visitor to return to the security checkpoint. The ASR was equipped with a nonlethal device and the robot used this device on one of the three visitors (a research confederate). Manipulations of reliability and stated social intent of the ASR were used in a 2 × 4 between subjects design (N = 320).

RESULTS

Reliability influenced trust and trustworthiness. Stated social intent influenced trustworthiness. Participants reported being more favorable toward use of the ASR in military contexts versus public contexts.

CONCLUSION

The study demonstrated that reliability of the ASR and statements regarding the ASR's stated social intent are important considerations influencing the trust process (inclusive of intentions to be vulnerable and trustworthiness perceptions).

APPLICATION

If robotic systems are authorized to use force against a human, public acceptance may be increased with availability of the intent-based programming of the robot and whether or not the robot's decision was reliable.

Human Performance Benefits of The Automation Transparency Design Principle : Validation and Variation

OBJECTIVE

Test the automation transparency design principle using a full-scope nuclear power plant simulator.

BACKGROUND

Automation transparency is a long-held human factors design principle espousing that the responsibilities, capabilities, goals, activities, and/or effects of automation should be directly observable in the human-system interface. The anticipated benefits of transparency include more effective reliance, more appropriate trust, better understanding, and greater user satisfaction. Transparency has enjoyed a recent upsurge in use in the context of human interaction with agent-oriented automation.

METHOD

Three full-scope nuclear power plant simulator studies were conducted with licensed operating crews. In the first two experiments, transparency was implemented for interlocks, controllers, limitations, protections, and automatic programs that operate at the local component level of the plant. In the third experiment, procedure automation assumed control of plant operations and was represented in dedicated agent displays.

RESULTS

Results from Experiments 1 and 2 appear to validate the human performance benefits of automation transparency for automation at the component level. However, Experiment 3 failed to replicate these findings for automation that assumed control for executing procedural actions.

CONCLUSION

Automation transparency appears to yield expected benefits for component-level automation, but caution is warranted in generalizing the design principle to agent-oriented automation.

APPLICATION

The automation transparency design principle may offer a powerful means of compensating for the detrimental impacts of hidden automation influence at the component level of complex systems. However, system developers should exercise caution in assuming that the principle extends to agent-oriented automation.

Brokerbot: A Cryptocurrency Chatbot in the Social-technical Gap of Trust

Cryptocurrencies are proliferating as instantiations of blockchain, which is a transparent, distributed ledger technology for validating transactions. Blockchain is thus said to embed trust in its technical design. Yet, blockchain’s technical promise of trust is not fulfilled when applied to the cryptocurrency ecosystem due to many social challenges stakeholders experience. By investigating a cryptocurrency chatbot (Brokerbot) that distributed information on cryptocurrency news and investments, we explored social tensions of trust between stakeholders, namely the bot’s developers, users, and the bot itself. We found that trust in Brokerbot and in the cryptocurrency ecosystem are two conjoined, but separate challenges that users and developers approached in different ways. We discuss the challenging, dual-role of a Brokerbot as an object of trust as a chatbot while simultaneously being a mediator of trust in cryptocurrency, which exposes the social-technical gap of trust. Lastly, we elaborate on trust as a negotiated social process that people shape and are shaped by through emerging ecologies of interlinked technologies like blockchain and conversational interfaces.

Effect of automation transparency in the management of multiple unmanned vehicles

Automated decision aids can undoubtedly benefit system performance, but have the potential to provide incorrect advice, creating the possibility for automation disuse or misuse. This paper examined the extent to which increased automation transparency could improve the accuracy of automation use in a simulation of unmanned vehicle control. Participants were required to assign the best unmanned vehicle to complete missions. An automated recommender system provided advice but was not always reliable. Three levels of automation transparency were manipulated between-participants. Providing transparency regarding the reasoning underlying automated recommendations improved the accuracy of automation use, with no cost to decision time or subjective workload. Higher transparency in the form of projected decision outcome visualizations led to faster decisions, but less accurate automation use and an automation bias. Implications for the design of transparent interfaces to improve human-autonomy teaming outcomes in time-pressured environments are discussed.

The reliability and transparency bases of trust in human-swarm interaction: principles and implications

Automation reliability and transparency are key factors for trust calibration and as such can have distinct effects on human reliance behaviour and mission performance. One question that remains unexplored is: what are the implications of reliability and transparency on trust calibration for human-swarm interaction? We investigate this research question in the context of human-swarm interaction, as swarm systems are becoming more popular for their robustness and versatility. Thirty-two participants performed swarm-based tasks under different reliability and transparency conditions. The results indicate that trust, whether it is reliability- or transparency-based, indicates high reliance rates and shorter response times. Reliability-based trust is negatively correlated with correct rejection rates while transparency-based trust is positively correlated with these rates. We conclude that reliability and transparency have distinct effects on trust calibration. Practitioner Summary: Reliability and transparency have distinct effects on trust calibration. Findings from our human experiments suggest that transparency is a necessary design requirement if and when humans need to be involved in the decision-loop of human-swarm systems, especially when swarm reliability is high. Abbreviations: HRI: human-robot interaction; IOS: inter-organisational systems; LMM: liner mixed models; MANOVA: multivariate analysis of variance; UxV: heterogeneous unmanned vehicles; UAV: unmanned aerial vehicle.

Trust and Distrust of Automated Parking in a Tesla Model X

OBJECTIVE

The present study aims to evaluate driver intervention behaviors during a partially automated parking task.

BACKGROUND

Cars with partially automated parking features are becoming widely available. Although recent research explores the use of automation features in partially automated cars, none have focused on partially automated parking. Recent incidents and research have demonstrated that drivers sometimes use partially automated features in unexpected, inefficient, and harmful ways.

METHOD

Participants completed a series of partially automated parking trials with a Tesla Model X and their behavioral interventions were recorded. Participants also completed a risk-taking behavior test and a post-experiment questionnaire that included questions about trust in the system, likelihood of using the Autopark feature, and preference for either the partially automated parking feature or self-parking.

RESULTS

Initial intervention rates were over 50%, but declined steeply in later trials. Responses to open-ended questions revealed that once participants understood what the system was doing, they were much more likely to trust it. Trust in the partially automated parking feature was predicted by a model including risk-taking behaviors, self-confidence, self-reported number of errors committed by the Tesla, and the proportion of trials in which the driver intervened.

CONCLUSION

Using partially automated parking with little knowledge of its workings can lead to high degree of initial distrust. Repeated exposure of partially automated features to drivers can greatly increase their use.

APPLICATION

Short tutorials and brief explanations of the workings of partially automated features may greatly improve trust in the system when drivers are first introduced to partially automated systems.

Adaptive trust calibration for human-AI collaboration

Safety and efficiency of human-AI collaboration often depend on how humans could appropriately calibrate their trust towards the AI agents. Over-trusting the autonomous system sometimes causes serious safety issues. Although many studies focused on the importance of system transparency in keeping proper trust calibration, the research in detecting and mitigating improper trust calibration remains very limited. To fill these research gaps, we propose a method of adaptive trust calibration that consists of a framework for detecting the inappropriate calibration status by monitoring the user’s reliance behavior and cognitive cues called “trust calibration cues” to prompt the user to reinitiate trust calibration. We evaluated our framework and four types of trust calibration cues in an online experiment using a drone simulator. A total of 116 participants performed pothole inspection tasks by using the drone’s automatic inspection, the reliability of which could fluctuate depending upon the weather conditions. The participants needed to decide whether to rely on automatic inspection or to do the inspection manually. The results showed that adaptively presenting simple cues could significantly promote trust calibration during over-trust.

Adapt, Explain, Engage—A Study on How Social Robots Can Scaffold Second-language Learning of Children

Social robots are increasingly applied to support children’s learning, but how a robot can foster (or may hinder) learning is still not fully clear. One technique used by teachers is scaffolding, temporarily assisting learners to achieve new skills or levels of understanding they would not reach on their own. We ask if and how a social robot can be utilized to scaffold second-language learning of children at kindergarten age (4--7 years). Specifically, we explore an adapt-and-explain scaffolding strategy in which a robot acts as a peer-like tutor who dynamically adapts its behavior or the learning tasks to the cognitive and affective state of the child, and provides verbal explanations of these adaptations. An evaluation study with 40 children shows that children benefit from the learning adaptation and that the explanations have a positive effect especially for slower learners. Further, in 76% of all cases the robot managed to “re-engage” children who started to disengage from the learning interaction, helping them to achieve an overall higher learning gain. These findings demonstrate that a social robot equipped with suitable scaffolding mechanisms can increase engagement and learning, especially when being adaptive to the individual behavior and states of a child learner.