1
|
Khanna R, Dodge J, Anderson A, Dikkala R, Irvine J, Shureih Z, Lam KH, Matthews CR, Lin Z, Kahng M, Fern A, Burnett M. Finding AI’s Faults with AAR/AI: An Empirical Study. ACM T INTERACT INTEL 2022. [DOI: 10.1145/3487065] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/01/2022]
Abstract
Would you allow an AI agent to make decisions on your behalf? If the answer is “not always,” the next question becomes “in what circumstances”? Answering this question requires human users to be able to assess an AI agent—and not just with overall pass/fail assessments or statistics. Here users need to be able to
localize
an agent’s bugs so that they can determine when they are willing to rely on the agent and when they are not. After-Action Review for AI (AAR/AI), a new AI assessment process for integration with Explainable AI systems, aims to support human users in this endeavor, and in this article we empirically investigate AAR/AI’s effectiveness with domain-knowledgeable users. Our results show that AAR/AI participants not only located significantly
more
bugs than non-AAR/AI participants did (i.e., showed greater recall) but also located them more
precisely
(i.e., with greater precision). In fact, AAR/AI participants outperformed non-AAR/AI participants on every bug and were, on average, almost six times as likely as non-AAR/AI participants to find any particular bug. Finally, evidence suggests that incorporating labeling into the AAR/AI process may encourage domain-knowledgeable users to abstract above individual instances of bugs; we hypothesize that doing so may have contributed further to AAR/AI participants’ effectiveness.
Collapse
Affiliation(s)
| | | | | | | | - Jed Irvine
- Oregon State University, Corvallis, OR, USA
| | | | - Kin-Ho Lam
- Oregon State University, Corvallis, OR, USA
| | | | | | | | - Alan Fern
- Oregon State University, Corvallis, OR, USA
| | | |
Collapse
|
2
|
Dodge J, Khanna R, Irvine J, Lam KH, Mai T, Lin Z, Kiddle N, Newman E, Anderson A, Raja S, Matthews C, Perdriau C, Burnett M, Fern A. After-Action Review for AI (AAR/AI). ACM T INTERACT INTEL 2021. [DOI: 10.1145/3453173] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/28/2023]
Abstract
Explainable AI is growing in importance as AI pervades modern society, but few have studied how explainable AI can directly support people trying to
assess
an AI agent. Without a rigorous process, people may approach assessment in ad hoc ways—leading to the possibility of wide variations in assessment of the same agent due only to variations in their processes. AAR, or After-Action Review, is a method some military organizations use to assess human agents, and it has been validated in many domains. Drawing upon this strategy, we derived an After-Action Review for AI (AAR/AI), to organize ways people assess reinforcement learning agents in a sequential decision-making environment. We then investigated what AAR/AI brought to human assessors in two qualitative studies. The first investigated AAR/AI to gather formative information, and the second built upon the results, and also varied the type of explanation (model-free vs. model-based) used in the AAR/AI process. Among the results were the following: (1) participants reporting that AAR/AI helped to
organize their thoughts
and
think logically
about the agent, (2) AAR/AI encouraged participants to reason about the agent from a
wide range of perspectives
, and (3) participants were able to leverage AAR/AI with the model-based explanations to
falsify
the agent’s predictions.
Collapse
Affiliation(s)
| | | | | | | | | | | | | | | | | | - Sai Raja
- Oregon State University, Corvallis, OR
| | | | | | | | - Alan Fern
- Oregon State University, Corvallis, OR
| |
Collapse
|
3
|
Dynamic exploration designs for graphical models using clustering with applications to petroleum exploration. Knowl Based Syst 2014. [DOI: 10.1016/j.knosys.2013.08.020] [Citation(s) in RCA: 8] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/23/2022]
|
4
|
Chen KH. Dynamic randomization and domain knowledge in Monte-Carlo Tree Search for Go knowledge-based systems. Knowl Based Syst 2012. [DOI: 10.1016/j.knosys.2011.08.007] [Citation(s) in RCA: 9] [Impact Index Per Article: 0.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/24/2022]
|
5
|
Pao HK, Fadlil J, Lin HY, Chen KT. Trajectory analysis for user verification and recognition. Knowl Based Syst 2012. [DOI: 10.1016/j.knosys.2012.03.008] [Citation(s) in RCA: 5] [Impact Index Per Article: 0.4] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/28/2022]
|
7
|
Limited-Damage A*: A path search algorithm that considers damage as a feasibility criterion. Knowl Based Syst 2011. [DOI: 10.1016/j.knosys.2010.12.009] [Citation(s) in RCA: 37] [Impact Index Per Article: 2.8] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
|