Towards Ethical AI: Empirically Investigating Dimensions of AI Ethics, Trust Repair, and Performance in Human-AI Teaming

OBJECTIVE

Determining the efficacy of two trust repair strategies (apology and denial) for trust violations of an ethical nature by an autonomous teammate.

BACKGROUND

While ethics in human-AI interaction is extensively studied, little research has investigated how decisions with ethical implications impact trust and performance within human-AI teams and their subsequent repair.

METHOD

Forty teams of two participants and one autonomous teammate completed three team missions within a synthetic task environment. The autonomous teammate made an ethical or unethical action during each mission, followed by an apology or denial. Measures of individual team trust, autonomous teammate trust, human teammate trust, perceived autonomous teammate ethicality, and team performance were taken.

RESULTS

Teams with unethical autonomous teammates had significantly lower trust in the team and trust in the autonomous teammate. Unethical autonomous teammates were also perceived as substantially more unethical. Neither trust repair strategy effectively restored trust after an ethical violation, and autonomous teammate ethicality was not related to the team score, but unethical autonomous teammates did have shorter times.

CONCLUSION

Ethical violations significantly harm trust in the overall team and autonomous teammate but do not negatively impact team score. However, current trust repair strategies like apologies and denials appear ineffective in restoring trust after this type of violation.

APPLICATION

This research highlights the need to develop trust repair strategies specific to human-AI teams and trust violations of an ethical nature.

Communication Strategies in Human-Autonomy Teams During Technological Failures

OBJECTIVE

This study examines low-, medium-, and high-performing Human-Autonomy Teams' (HATs') communication strategies during various technological failures that impact routine communication strategies to adapt to the task environment.

BACKGROUND

Teams must adapt their communication strategies during dynamic tasks, where more successful teams make more substantial adaptations. Adaptations in communication strategies may explain how successful HATs overcome technological failures. Further, technological failures of variable severity may alter communication strategies of HATs at different performance levels in their attempts to overcome each failure.

METHOD

HATs in a Remotely Piloted Aircraft System-Synthetic Task Environment (RPAS-STE), involving three team members, were tasked with photographing targets. Each triad had two randomly assigned participants in navigator and photographer roles, teaming with an experimenter who simulated an AI pilot in a Wizard of Oz paradigm. Teams encountered two different technological failures, automation and autonomy, where autonomy failures were more challenging to overcome.

RESULTS

High-performing HATs calibrated their communication strategy to the complexity of the different failures better than medium- and low-performing teams. Further, HATs adjusted their communication strategies over time. Finally, only the most severe failures required teams to increase the efficiency of their communication.

CONCLUSION

HAT effectiveness under degraded conditions depends on the type of communication strategies enacted by the team. Previous findings from studies of all-human teams apply here; however, novel results suggest information requests are particularly important to HAT success during failures.

APPLICATION

Understanding the communication strategies of HATs under degraded conditions can inform training protocols to help HATs overcome failures.

The Impact of Training on Human-Autonomy Team Communications and Trust Calibration

OBJECTIVE

This work examines two human-autonomy team (HAT) training approaches that target communication and trust calibration to improve team effectiveness under degraded conditions.

BACKGROUND

Human-autonomy teaming presents challenges to teamwork, some of which may be addressed through training. Factors vital to HAT performance include communication and calibrated trust.

METHOD

Thirty teams of three, including one confederate acting as an autonomous agent, received either entrainment-based coordination training, trust calibration training, or control training before executing a series of missions operating a simulated remotely piloted aircraft. Automation and autonomy failures simulating degraded conditions were injected during missions, and measures of team communication, trust, and task efficiency were collected.

RESULTS

Teams receiving coordination training had higher communication anticipation ratios, took photos of targets faster, and overcame more autonomy failures. Although autonomy failures were introduced in all conditions, teams receiving the calibration training reported that their overall trust in the agent was more robust over time. However, they did not perform better than the control condition.

CONCLUSIONS

Training based on entrainment of communications, wherein introduction of timely information exchange through one team member has lasting effects throughout the team, was positively associated with improvements in HAT communications and performance under degraded conditions. Training that emphasized the shortcomings of the autonomous agent appeared to calibrate expectations and maintain trust.

APPLICATIONS

Team training that includes an autonomous agent that models effective information exchange may positively impact team communication and coordination. Training that emphasizes the limitations of an autonomous agent may help calibrate trust.

Teaming With a Synthetic Teammate: Insights into Human-Autonomy Teaming

Objective Three different team configurations are compared with the goal of better understanding human-autonomy teaming (HAT). Background Although an extensive literature on human-automation interaction exists, much less is known about HAT in which humans and autonomous agents interact as coordinated units. Further research must be conducted to better understand how all-human teams compare to HAT. Methods In an unmanned aerial system (UAS) context, a comparison was made among three types of three-member teams: (1) synthetic teams in which the pilot role is assigned to a synthetic teammate, (2) control teams in which the pilot was an inexperienced human, and (3) experimenter teams in which an experimenter served as an experienced pilot. Ten of each type of team participated. Measures of team performance, target processing efficiency, team situation awareness, and team verbal behaviors were analyzed. Results Synthetic teams performed as well at the mission level as control (all human) teams but processed targets less efficiently. Experimenter teams performed better across all other measures compared to control and synthetic teams. Conclusion Though there is potential for a synthetic agent to function as a full-fledged teammate, further advances in autonomy are needed to improve team-level dynamics in HAT teams. Application This research contributes to our understanding of how to make autonomy a good team player.

Investigating Team Coordination in Baseball Using a Novel Joint Decision Making Paradigm

A novel joint decision making paradigm for assessing team coordination was developed and tested using baseball infielders. Balls launched onto an infield at different trajectories were filmed using four video cameras that were each placed at one of the typical positions of the four infielders. Each participant viewed temporally occluded videos for one of the four positions and were asked to say either “ball” if they would attempt to field it or the name of the bag that they would cover. The evaluation of two experienced coaches was used to assign a group coordination score for each trajectory and group decision times were calculated. Thirty groups of 4 current college baseball players were: (i) teammates (players from same team/view from own position), (ii) non-teammates (players from different teams/view from own position), or (iii) scrambled teammates (players from same team/view not from own position). Teammates performed significantly better (i.e., faster and more coordinated decisions) than the other two groups, whereas scrambled teammates performed significantly better than non-teammates. These findings suggest that team coordination is achieved through both experience with one’s teammates’ responses to particular events (e.g., a ball hit up the middle) and one’s own general action capabilities (e.g., running speed). The sensitivity of our joint decision making paradigm to group makeup provides support for its use as a method for studying team coordination.

Identification of the Emplacement of Improvised Explosive Devices by Experienced Mission Payload Operators

Improvised Explosive Devices (IEDs) have become one of the deadliest threats to military personnel, resulting in over 50% of American combat casualties in Iraq and Afghanistan. Identification of IED emplacement is conducted by mission payload operators (MPOs). Yet, experienced MPOs are limited in number, making MPO training a critical intervention. In this article, we implement a Cognitive Engineering Based on Expert Skill methodology to better understand how experienced MPOs identify the emplacement of IEDs for the purposes of improving training. First, expert knowledge was elicited through interviews and questionnaires to identify the types of perceptual cues used and how these cues are cognitively processed. Results indicate that there are many different static and dynamic cues that interact with each other over time and space. Using data from the interviews and questionnaires, an empirically grounded framework is presented that explains the cognitive process of IED emplacement detection. Using the overall findings and the framework, IED emplacement training scenarios were developed and built into a simulation.

Synthetic Teammate Communication and Coordination With Humans

A synthetic teammate based on ACT-R cognitive architecture has been developed to function as an Air Vehicle Operator in the context of a three-agent Unmanned Aerial Vehicle (UAV) ground control team taking part in studies in a Synthetic Task Environment (STE). In order for the synthetic teammate to function as team player with human teammates, it needs to skillfully handle the subtleties of team communication and coordination. Data from early synthetic teammate interactions with two human teammates are presented here to illustrate team communication and coordination challenges for the synthetic teammate. In turn, the synthetic teammate limitations have highlighted the intricacies involved in effective teamwork. Communication, though a terrifically challenging problem in itself, is only a foundation for coordinated teamwork or interacting as a team player.

Team Cognition As a Means to Improve Care Delivery in Critically Ill Patients With Cancer After Hematopoietic Cell Transplantation

Hematopoietic cell transplantation (HCT) is an important and complex treatment modality for a variety of hematologic malignancies and some solid tumors. Although outcomes of patients who have undergone HCT and require care in intensive care units (ICUs) have improved over time, mortality rates remain high and there are significant associated costs. Lack of a team-based approach to care, especially during critical illness, is detrimental to patient autonomy and satisfaction, and to team morale, ultimately leading to poor quality of care. In this manuscript, we describe the case of a patient who had undergone HCT and was in the ICU setting, where inconsistent team interaction among the various stakeholders delivering care resulted in a lack of shared goals and poor outcomes. Team cognition is cognitive processing at the team level through interactions among team members and is reflected in dynamic communication and coordination behaviors. Although the patient received multidisciplinary care as needed in a medically complicated case, a lack of team cognition and, particularly, inconsistent communication among the dynamic teams caring for the patient, led to mixed messages being delivered with high-cost implications for the health-care system and the family. This article highlights concepts and recommendations that begin a necessary in-depth assessment of implications for clinical care and initiate a research agenda that examines the effects of team cognition on HCT teams, and, more generally, critical care of the patient with cancer.

The Synthetic Teammate as a Team Player in Command-and-Control Teams

Project overview. The current project is part of a larger effort that focuses on Human-Automation Teaming (HAT) interaction in the context of the development, integration, and validation of a computational cognitive model that acts as a full-fledged synthetic teammate for a three-agent Unmanned Aircraft System (UAS) ground control crew. Our most recent effort looked at team process and team performance within the HAT. In order to be considered a team player, the synthetic teammate must be able to communicate and coordinate with its human teammates and do so in a subtle manner (Demir et al., 2016). In this task, there were three different and interdependent team members: 1) Air Vehicle Operator (AVO) – controls the UAS’s heading, altitude, and airspeed; 2) Data Exploitation, Mission Planning, and Communications (DEMPC) – provides a dynamic flight plan as well as speed and altitude restrictions; and 3) Payload Operator (PLO) – monitors sensor equipment, negotiates with the AVO, and takes photographs of target waypoints. The communication within a three-agent UAS team occurred over a text-based communications system. In this research, there were three conditions which are differentiated by the AVO role: 1) the Synthetic - the synthetic teammate was assigned the AVO role; 2) the Control - the AVO was an inexperienced human participant; 3) the Experimenter - the AVO was one of the experimenters who was experienced with the task. The experimenter AVO asked questions of other team members to ensure timely and adaptive passing of information at target waypoints. In this current study, the coordination among the team members occurs at each target waypoint and requires a specific sequence of information passing for an optimum team performance (Cooke, Gorman, Duran, & Taylor, 2007): the information is provided by the DEMPC about the upcoming target waypoint to the AVO. After that, the PLO and the AVO negotiate regarding an appropriate altitude and airspeed for the target waypoints about required camera settings. Finally, the PLO sends feedback to other team members about the status of the target photo.

Method. Activities during this period included conducting an experiment to: 1) evaluate the synthetic teammate’s performance, and the HAT team performance in comparison to all human teams, 2) understand how team process differs between all human and human-synthetic teams and how this impacts performance, and 3) compare the human-synthetic teams and all human control teams to a team with a pilot that is experienced in pushing and pulling information across the team. For this experiment, participants were randomly assigned for the duration of the experiment. Within each of the five missions, teams were told to obtain as many “good” photos as possible while avoiding alarms and rule violations in less than 40 minutes. The overall focus of this paper is: team process that is comprised of eight verbal behaviors associated with team effectiveness; team performance that is a combination of mission variables, including the rate of successful target photographs, time spent in alarm and warning states (for each individual), and the critical waypoint acquisition rate; and target processing efficiency took into account the time spent inside a target waypoint to get a good photo.

Results and discussion. In general, findings indicate that synthetic AVOs perform more poorly than control AVOs in terms of team performance. Synthetic teams perform as well at the mission level as control teams. However, in terms of target processing efficiency, synthetic teams perform poorer than control teams. In terms of team process, synthetic teams demonstrate interaction patterns corresponding to more pulling of information than pushing with little change over time. In summary, these results indicate that there is a strong potential for using synthetic team member as a teammate in real world tasks and for training.