1
|
Lebrun B, Temtsin S, Vonasch A, Bartneck C. Detecting the corruption of online questionnaires by artificial intelligence. Front Robot AI 2024; 10:1277635. [PMID: 38371744 PMCID: PMC10869497 DOI: 10.3389/frobt.2023.1277635] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/15/2023] [Accepted: 12/04/2023] [Indexed: 02/20/2024] Open
Abstract
Online questionnaires that use crowdsourcing platforms to recruit participants have become commonplace, due to their ease of use and low costs. Artificial intelligence (AI)-based large language models (LLMs) have made it easy for bad actors to automatically fill in online forms, including generating meaningful text for open-ended tasks. These technological advances threaten the data quality for studies that use online questionnaires. This study tested whether text generated by an AI for the purpose of an online study can be detected by both humans and automatic AI detection systems. While humans were able to correctly identify the authorship of such text above chance level (76% accuracy), their performance was still below what would be required to ensure satisfactory data quality. Researchers currently have to rely on a lack of interest among bad actors to successfully use open-ended responses as a useful tool for ensuring data quality. Automatic AI detection systems are currently completely unusable. If AI submissions of responses become too prevalent, then the costs associated with detecting fraudulent submissions will outweigh the benefits of online questionnaires. Individual attention checks will no longer be a sufficient tool to ensure good data quality. This problem can only be systematically addressed by crowdsourcing platforms. They cannot rely on automatic AI detection systems and it is unclear how they can ensure data quality for their paying clients.
Collapse
Affiliation(s)
- Benjamin Lebrun
- School of Psychology, Speech, and Hearing, University of Canterbury, Christchurch, New Zealand
| | - Sharon Temtsin
- Department of Computer Science and Software Engineering, University of Canterbury, Christchurch, New Zealand
| | - Andrew Vonasch
- School of Psychology, Speech, and Hearing, University of Canterbury, Christchurch, New Zealand
| | - Christoph Bartneck
- Department of Computer Science and Software Engineering, University of Canterbury, Christchurch, New Zealand
| |
Collapse
|
2
|
Leichtmann B, Nitsch V, Mara M. Crisis Ahead? Why Human-Robot Interaction User Studies May Have Replicability Problems and Directions for Improvement. Front Robot AI 2022; 9:838116. [PMID: 35360497 PMCID: PMC8961736 DOI: 10.3389/frobt.2022.838116] [Citation(s) in RCA: 4] [Impact Index Per Article: 2.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Journal Information] [Subscribe] [Scholar Register] [Received: 12/17/2021] [Accepted: 02/17/2022] [Indexed: 11/17/2022] Open
Abstract
There is a confidence crisis in many scientific disciplines, in particular disciplines researching human behavior, as many effects of original experiments have not been replicated successfully in large-scale replication studies. While human-robot interaction (HRI) is an interdisciplinary research field, the study of human behavior, cognition and emotion in HRI plays also a vital part. Are HRI user studies facing the same problems as other fields and if so, what can be done to overcome them? In this article, we first give a short overview of the replicability crisis in behavioral sciences and its causes. In a second step, we estimate the replicability of HRI user studies mainly 1) by structural comparison of HRI research processes and practices with those of other disciplines with replicability issues, 2) by systematically reviewing meta-analyses of HRI user studies to identify parameters that are known to affect replicability, and 3) by summarizing first replication studies in HRI as direct evidence. Our findings suggest that HRI user studies often exhibit the same problems that caused the replicability crisis in many behavioral sciences, such as small sample sizes, lack of theory, or missing information in reported data. In order to improve the stability of future HRI research, we propose some statistical, methodological and social reforms. This article aims to provide a basis for further discussion and a potential outline for improvements in the field.
Collapse
Affiliation(s)
- Benedikt Leichtmann
- LIT Robopsychology Lab, Johannes Kepler University Linz, Linz, Austria
- *Correspondence: Benedikt Leichtmann,
| | - Verena Nitsch
- Institute of Industrial Engineering and Ergonomics, RWTH Aachen University, Aachen, Germany
| | - Martina Mara
- LIT Robopsychology Lab, Johannes Kepler University Linz, Linz, Austria
| |
Collapse
|
3
|
Fraune MR, Leite I, Karatas N, Amirova A, Legeleux A, Sandygulova A, Neerincx A, Dilip Tikas G, Gunes H, Mohan M, Abbasi NI, Shenoy S, Scassellati B, de Visser EJ, Komatsu T. Lessons Learned About Designing and Conducting Studies From HRI Experts. Front Robot AI 2022; 8:772141. [PMID: 35155588 PMCID: PMC8832512 DOI: 10.3389/frobt.2021.772141] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/07/2021] [Accepted: 11/18/2021] [Indexed: 01/04/2023] Open
Abstract
The field of human-robot interaction (HRI) research is multidisciplinary and requires researchers to understand diverse fields including computer science, engineering, informatics, philosophy, psychology, and more disciplines. However, it is hard to be an expert in everything. To help HRI researchers develop methodological skills, especially in areas that are relatively new to them, we conducted a virtual workshop, Workshop Your Study Design (WYSD), at the 2021 International Conference on HRI. In this workshop, we grouped participants with mentors, who are experts in areas like real-world studies, empirical lab studies, questionnaire design, interview, participatory design, and statistics. During and after the workshop, participants discussed their proposed study methods, obtained feedback, and improved their work accordingly. In this paper, we present 1) Workshop attendees’ feedback about the workshop and 2) Lessons that the participants learned during their discussions with mentors. Participants’ responses about the workshop were positive, and future scholars who wish to run such a workshop can consider implementing their suggestions. The main contribution of this paper is the lessons learned section, where the workshop participants contributed to forming this section based on what participants discovered during the workshop. We organize lessons learned into themes of 1) Improving study design for HRI, 2) How to work with participants - especially children -, 3) Making the most of the study and robot’s limitations, and 4) How to collaborate well across fields as they were the areas of the papers submitted to the workshop. These themes include practical tips and guidelines to assist researchers to learn about fields of HRI research with which they have limited experience. We include specific examples, and researchers can adapt the tips and guidelines to their own areas to avoid some common mistakes and pitfalls in their research.
Collapse
Affiliation(s)
- Marlena R. Fraune
- Intergroup Human-Robot Interaction (iHRI) Lab, Department of Psychology, New Mexico State University, Las Cruces, NM, United States
- *Correspondence: Marlena R. Fraune,
| | - Iolanda Leite
- Division of Robotics, Perception, and Learning (RPL), School of Electrical Engineering and Computer Science, KTH Royal Institute of Technology, Stockholm, Sweden
| | - Nihan Karatas
- Human-Machine Interaction (HMI) and Human Characteristics Research Division, Institutes of Innovation for Future Society, Nagoya University, Nagoya, Japan
| | - Aida Amirova
- Department of Robotics and Mechatronics, School of Engineering and Digital Sciences, Nazarbayev University, Nur-Sultan, Kazakhstan
| | - Amélie Legeleux
- Lab-STICC, University of South Brittany, CNRS UMR 6285, Brest, France
| | - Anara Sandygulova
- Department of Robotics and Mechatronics, School of Engineering and Digital Sciences, Nazarbayev University, Nur-Sultan, Kazakhstan
| | - Anouk Neerincx
- Lab-STICC, University of South Brittany, CNRS UMR 6285, Brest, France
| | - Gaurav Dilip Tikas
- Strategy, Innovation and Entrepreneurship Area, Institute of Management Technology, Ghaziabad, India
| | - Hatice Gunes
- Affective Intelligence and Robotics Lab, Department of Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom
| | - Mayumi Mohan
- Haptic Intelligence Department, Max Planck Institute for Intelligent Systems, Stuttgart, Germany
| | - Nida Itrat Abbasi
- Affective Intelligence and Robotics Lab, Department of Computer Science and Technology, University of Cambridge, Cambridge, United Kingdom
| | - Sudhir Shenoy
- Human-AI Technology Lab, Computer Engineering Program, University of Virginia, Charlottesville, VA, United States
| | - Brian Scassellati
- Social Robotics Lab, Department of Computer Science, Yale University, New Haven, CT, United States
| | - Ewart J. de Visser
- Warfighter Effectiveness Research Center, U.S. Air Force Academy, Colorado Springs, CO, United States
| | - Takanori Komatsu
- Department of Frontier Media Science, School of Interdisciplinary Mathematical Science, Meiji University, Tokyo, Japan
| |
Collapse
|
4
|
Ostrowski AK, Fu J, Zygouras V, Park HW, Breazeal C. Speed Dating with Voice User Interfaces: Understanding How Families Interact and Perceive Voice User Interfaces in a Group Setting. Front Robot AI 2022; 8:730992. [PMID: 35141285 PMCID: PMC8819708 DOI: 10.3389/frobt.2021.730992] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/25/2021] [Accepted: 11/03/2021] [Indexed: 11/13/2022] Open
Abstract
As voice-user interfaces (VUIs), such as smart speakers like Amazon Alexa or social robots like Jibo, enter multi-user environments like our homes, it is critical to understand how group members perceive and interact with these devices. VUIs engage socially with users, leveraging multi-modal cues including speech, graphics, expressive sounds, and movement. The combination of these cues can affect how users perceive and interact with these devices. Through a set of three elicitation studies, we explore family interactions (N = 34 families, 92 participants, ages 4–69) with three commercially available VUIs with varying levels of social embodiment. The motivation for these three studies began when researchers noticed that families interacted differently with three agents when familiarizing themselves with the agents and, therefore, we sought to further investigate this trend in three subsequent studies designed as a conceptional replication study. Each study included three activities to examine participants’ interactions with and perceptions of the three VUIS in each study, including an agent exploration activity, perceived personality activity, and user experience ranking activity. Consistent for each study, participants interacted significantly more with an agent with a higher degree of social embodiment, i.e., a social robot such as Jibo, and perceived the agent as more trustworthy, having higher emotional engagement, and having higher companionship. There were some nuances in interaction and perception with different brands and types of smart speakers, i.e., Google Home versus Amazon Echo, or Amazon Show versus Amazon Echo Spot between the studies. In the last study, a behavioral analysis was conducted to investigate interactions between family members and with the VUIs, revealing that participants interacted more with the social robot and interacted more with their family members around the interactions with the social robot. This paper explores these findings and elaborates upon how these findings can direct future VUI development for group settings, especially in familial settings.
Collapse
Affiliation(s)
- Anastasia K. Ostrowski
- Media Lab, Massachusetts Institute of Technology, Cambridge, MA, United States
- *Correspondence: Anastasia K. Ostrowski,
| | - Jenny Fu
- Information Science, Cornell University, Ithaca, NY, United States
| | - Vasiliki Zygouras
- Media Lab, Massachusetts Institute of Technology, Cambridge, MA, United States
| | - Hae Won Park
- Media Lab, Massachusetts Institute of Technology, Cambridge, MA, United States
| | - Cynthia Breazeal
- Media Lab, Massachusetts Institute of Technology, Cambridge, MA, United States
| |
Collapse
|