1
|
Wang XM, Southwick D, Robinson I, Nitsche M, Resch G, Mazalek A, Welsh TN. Prolonged exposure to mixed reality alters task performance in the unmediated environment. Sci Rep 2024; 14:18938. [PMID: 39147910 PMCID: PMC11327334 DOI: 10.1038/s41598-024-69116-w] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/04/2023] [Accepted: 07/31/2024] [Indexed: 08/17/2024] Open
Abstract
The popularity of mixed reality (MR) technologies, including virtual (VR) and augmented (AR) reality, have advanced many training and skill development applications. If successful, these technologies could be valuable for high-impact professional training, like medical operations or sports, where the physical resources could be limited or inaccessible. Despite MR's potential, it is still unclear whether repeatedly performing a task in MR would affect performance in the same or related tasks in the physical environment. To investigate this issue, participants executed a series of visually-guided manual pointing movements in the physical world before and after spending one hour in VR or AR performing similar movements. Results showed that, due to the MR headsets' intrinsic perceptual geometry, movements executed in VR were shorter and movements executed in AR were longer than the veridical Euclidean distance. Crucially, the sensorimotor bias in MR conditions also manifested in the subsequent post-test pointing task; participants transferring from VR initially undershoot whereas those from AR overshoot the target in the physical environment. These findings call for careful consideration of MR-based training because the exposure to MR may perturb the sensorimotor processes in the physical environment and negatively impact performance accuracy and transfer of training from MR to UR.
Collapse
Affiliation(s)
- Xiaoye Michael Wang
- Faculty of Kinesiology & Physical Education, University of Toronto, Toronto, ON, Canada.
| | - Daniel Southwick
- Synaesthetic Media Lab, Toronto Metropolitan University, Toronto, ON, Canada
| | - Ian Robinson
- Synaesthetic Media Lab, Toronto Metropolitan University, Toronto, ON, Canada
| | - Michael Nitsche
- Ivan Allen College of Liberal Arts, Georgia Institute of Technology, Atlanta, GA, USA
| | - Gabby Resch
- Faculty of Business and Information Technology, Ontario Tech University, Oshawa, ON, Canada
| | - Ali Mazalek
- Synaesthetic Media Lab, Toronto Metropolitan University, Toronto, ON, Canada
| | - Timothy N Welsh
- Faculty of Kinesiology & Physical Education, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
2
|
Molnar J, Agrawal V, Chernova S. Clustering user preferences for personalized teleoperation control schemes via trajectory similarity analysis. Front Robot AI 2024; 11:1330812. [PMID: 38654756 PMCID: PMC11035831 DOI: 10.3389/frobt.2024.1330812] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/31/2023] [Accepted: 03/18/2024] [Indexed: 04/26/2024] Open
Abstract
Successful operation of a teleoperated robot depends on a well-designed control scheme to translate human motion into robot motion; however, a single control scheme may not be suitable for all users. On the other hand, individual personalization of control schemes may be infeasible for designers to produce. In this paper, we present a method by which users may be classified into groups with mutually compatible control scheme preferences. Users are asked to demonstrate freehand motions to control a simulated robot in a virtual reality environment. Hand pose data is captured and compared with other users using SLAM trajectory similarity analysis techniques. The resulting pairwise trajectory error metrics are used to cluster participants based on their control motions, without foreknowledge of the number or types of control scheme preferences that may exist. The clusters identified for two different robots shows that a small number of clusters form stably for each case, each with its own control scheme paradigm. Survey data from participants validates that the clusters identified through this method correspond to the participants' control scheme rationales, and also identify nuances in participant control scheme descriptions that may not be obvious to designers relying only on participant explanations of their preferences.
Collapse
Affiliation(s)
- Jennifer Molnar
- Woodruff School of Mechanical Engineering, Georgia Institute of Technology, Atlanta, GA, United States
| | - Varun Agrawal
- School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA, United States
| | - Sonia Chernova
- School of Interactive Computing, Georgia Institute of Technology, Atlanta, GA, United States
| |
Collapse
|
3
|
Cesari V, D’Aversa S, Piarulli A, Melfi F, Gemignani A, Menicucci D. Sense of Agency and Skills Learning in Virtual-Mediated Environment: A Systematic Review. Brain Sci 2024; 14:350. [PMID: 38672002 PMCID: PMC11048251 DOI: 10.3390/brainsci14040350] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/21/2024] [Revised: 03/27/2024] [Accepted: 03/29/2024] [Indexed: 04/28/2024] Open
Abstract
Agency is central to remote actions, and it may enhance skills learning due to a partial overlap between brain structures and networks, the promotion of confidence towards a telemanipulator, and the feeling of congruence of the motor choice to the motor plan. We systematically reviewed studies aiming to verify the role of agency in improving learning. Fifteen studies were selected from MEDLINE and Scopus®. When a mismatch is introduced between observed and performed actions, the decrease in agency and learning is proportional to the intensity of the mismatch, which is due to greater interference with the motor programming. Thanks to multisensory integration, agency and learning benefit both from sensory and performance feedback and from the timing of feedback based on control at the goal level or the perceptual-motor level. This work constitutes a bedrock for professional teleoperation settings (e.g., robotic surgery), with particular reference to the role of agency in performing complex tasks with remote control.
Collapse
Affiliation(s)
- Valentina Cesari
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, 56126 Pisa, Italy; (V.C.); (S.D.); (A.P.); (F.M.); (A.G.)
| | - Sveva D’Aversa
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, 56126 Pisa, Italy; (V.C.); (S.D.); (A.P.); (F.M.); (A.G.)
| | - Andrea Piarulli
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, 56126 Pisa, Italy; (V.C.); (S.D.); (A.P.); (F.M.); (A.G.)
| | - Franca Melfi
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, 56126 Pisa, Italy; (V.C.); (S.D.); (A.P.); (F.M.); (A.G.)
| | - Angelo Gemignani
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, 56126 Pisa, Italy; (V.C.); (S.D.); (A.P.); (F.M.); (A.G.)
- Clinical Psychology Branch, Azienda Ospedaliero-Universitaria Pisana, 56126 Pisa, Italy
| | - Danilo Menicucci
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, 56126 Pisa, Italy; (V.C.); (S.D.); (A.P.); (F.M.); (A.G.)
| |
Collapse
|
4
|
Cesari V, Melfi F, Gemignani A, Menicucci D. Sensory substitution increases robotic surgical performance and sets the ground for a mediating role of the sense of embodiment: a systematic review. Heliyon 2023; 9:e21665. [PMID: 38027699 PMCID: PMC10656242 DOI: 10.1016/j.heliyon.2023.e21665] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 10/23/2022] [Revised: 09/17/2023] [Accepted: 10/25/2023] [Indexed: 12/01/2023] Open
Abstract
Sensory Substitution (SS) allows the elaboration of information via non preferential sensory modalities. This phenomenon occurs in robotic-assisted surgery (RAS), in which haptic feedback is lacking. It has been suggested that SS could sustain surgeons' proficiency by means of visual clues for inferring tactile information, that also promotes the feeling of haptic phantom sensations. A critical role in reaching a good performance in procedural tasks is also sustained by the Sense of Embodiment (SE), that is, the capacity to integrate objects into subjective bodily self-representation. As SE is enhanced by haptic sensations, we hypothesize a role of SS in promoting SE in RAS. Accordingly, the goal of this systematic review is to summarize the evidence pertaining the study of SS in RAS in order to highlight the impact on the performance, and to identify a mediating role of the SE in increasing dexterity in RAS. Eight studies selected from the MEDLINE and Scopus® databases met inclusion criteria for a qualitative synthesis. Results indicated that haptic to other modalities SS enhanced force consistency and accuracy, and decreased surgeon fatigue. Expert surgeons, as compared to novices, showed a better natural SS processing, testified by a proficient performance with and without SS aids. No studies investigated the mediating role of SE. These findings indicate that SS is subjected to learning and memory processes that help surgeons to rapidly derive haptic-correlates from visual clues, which are highly required for a good performance. Also, the higher ability of doing SS and the associated perception of haptic sensations might increase multisensory integration, which might sustain performance.
Collapse
Affiliation(s)
- Valentina Cesari
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, via Savi, 10, 56126, Pisa, Italy
| | - Franca Melfi
- Robotic Multispecialty Center for Surgery Robotic, Minimally Invasive Thoracic Surgery, University of Pisa, 56124 Pisa, Italy
| | - Angelo Gemignani
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, via Savi, 10, 56126, Pisa, Italy
| | - Danilo Menicucci
- Department of Surgical, Medical and Molecular Pathology and Critical Care Medicine, University of Pisa, via Savi, 10, 56126, Pisa, Italy
| |
Collapse
|
5
|
Vourvopoulos A, Fleury M, Tonin L, Perdikis S. Editorial: Neurotechnologies and brain-computer interaction for neurorehabilitation. FRONTIERS IN NEUROERGONOMICS 2023; 4:1203934. [PMID: 38234475 PMCID: PMC10790924 DOI: 10.3389/fnrgo.2023.1203934] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Key Words] [Track Full Text] [Subscribe] [Scholar Register] [Received: 04/11/2023] [Accepted: 06/08/2023] [Indexed: 01/19/2024]
Affiliation(s)
- Athanasios Vourvopoulos
- Department of Bioengineering, Institute for Systems and Robotics - Lisboa, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal
| | - Mathis Fleury
- Department of Bioengineering, Institute for Systems and Robotics - Lisboa, Instituto Superior Técnico, Universidade de Lisboa, Lisbon, Portugal
| | - Luca Tonin
- Department of Information Engineering, Università degli Studi di Padova, Padua, Italy
| | - Serafeim Perdikis
- Brain-Computer Interfaces and Neural Engineering Laboratory, School of Computer Science and Electronic Engineering, University of Essex, Colchester, United Kingdom
| |
Collapse
|
6
|
Morita T, Zhu Y, Aoyama T, Takeuchi M, Yamamoto K, Hasegawa Y. Auditory Feedback for Enhanced Sense of Agency in Shared Control. SENSORS (BASEL, SWITZERLAND) 2022; 22:9779. [PMID: 36560147 PMCID: PMC9787405 DOI: 10.3390/s22249779] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Track Full Text] [Figures] [Subscribe] [Scholar Register] [Received: 10/13/2022] [Revised: 12/01/2022] [Accepted: 12/10/2022] [Indexed: 06/17/2023]
Abstract
There is a growing need for robots that can be remotely controlled to perform tasks of one's own choice. However, the SoA (Sense of Agency: the sense of recognizing that the motion of an observed object is caused by oneself) is reduced because the subject of the robot motion is identified as external due to shared control. To address this issue, we aimed to suppress the decline in SoA by presenting auditory feedback that aims to blur the distinction between self and others. We performed the tracking task in a virtual environment under four different auditory feedback conditions, with varying levels of automation to manipulate the virtual robot gripper. Experimental results showed that the proposed auditory feedback suppressed the decrease in the SoA at a medium level of automation. It is suggested that our proposed auditory feedback could blur the distinction between self and others, and that the operator attributes the subject of the motion of the manipulated object to himself.
Collapse
Affiliation(s)
- Tomoya Morita
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya 464-8601, Japan
| | - Yaonan Zhu
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya 464-8601, Japan
| | - Tadayoshi Aoyama
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya 464-8601, Japan
| | - Masaru Takeuchi
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya 464-8601, Japan
| | - Kento Yamamoto
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya 464-8601, Japan
| | - Yasuhisa Hasegawa
- Department of Micro-Nano Mechanical Science and Engineering, Nagoya University, Nagoya 464-8601, Japan
| |
Collapse
|
7
|
Human-Robot Body Experience: An Artificial Intelligence Perspective. KUNSTLICHE INTELLIGENZ 2022. [DOI: 10.1007/s13218-022-00779-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/29/2022]
Abstract
AbstractHuman body experience is remarkably flexible, which enables us to integrate passive tools as well as intelligent robotic devices into our body representation. Accordingly, it can serve as a role model to make (assistive) robots interact seamlessly with their users or to provide (humanoid) robots with a human-like self-perception and behavior generation. This article discusses the potential of understanding human body experience and applying it to robotics. Particular focus is set on how to use artificial intelligence techniques and create intelligent artificial agents from insights about human body experience. The discussion is based on a summary of the author’s habilitation thesis and combines theoretical and experimental perspectives from psychology, cognitive science and neuroscience as well as computer science, engineering, and artificial intelligence. From this, it derives directions for future developments towards creating artificial body intelligence with human-like capabilities.
Collapse
|
8
|
Virtual Reality for Safe Testing and Development in Collaborative Robotics: Challenges and Perspectives. ELECTRONICS 2022. [DOI: 10.3390/electronics11111726] [Citation(s) in RCA: 2] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 11/17/2022]
Abstract
Collaborative robots (cobots) could help humans in tasks that are mundane, dangerous or where direct human contact carries risk. Yet, the collaboration between humans and robots is severely limited by the aspects of the safety and comfort of human operators. In this paper, we outline the use of extended reality (XR) as a way to test and develop collaboration with robots. We focus on virtual reality (VR) in simulating collaboration scenarios and the use of cobot digital twins. This is specifically useful in situations that are difficult or even impossible to safely test in real life, such as dangerous scenarios. We describe using XR simulations as a means to evaluate collaboration with robots without putting humans at harm. We show how an XR setting enables combining human behavioral data, subjective self-reports, and biosignals signifying human comfort, stress and cognitive load during collaboration. Several works demonstrate XR can be used to train human operators and provide them with augmented reality (AR) interfaces to enhance their performance with robots. We also provide a first attempt at what could become the basis for a human–robot collaboration testing framework, specifically for designing and testing factors affecting human–robot collaboration. The use of XR has the potential to change the way we design and test cobots, and train cobot operators, in a range of applications: from industry, through healthcare, to space operations.
Collapse
|
9
|
Connan M, Sierotowicz M, Henze B, Porges O, Albu-Schaeffer A, Roa M, Castellini C. Learning to teleoperate an upper-limb assistive humanoid robot for bimanual daily-living tasks. Biomed Phys Eng Express 2021; 8. [PMID: 34757953 DOI: 10.1088/2057-1976/ac3881] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/03/2021] [Accepted: 11/10/2021] [Indexed: 11/12/2022]
Abstract
Objective.Bimanual humanoid platforms for home assistance are nowadays available, both as academic prototypes and commercially. Although they are usually thought of as daily helpers for non-disabled users, their ability to move around, together with their dexterity, makes them ideal assistive devices for upper-limb disabled persons, too. Indeed, teleoperating a bimanual robotic platform via muscle activation could revolutionize the way stroke survivors, amputees and patients with spinal injuries solve their daily home chores. Moreover, with respect to direct prosthetic control, teleoperation has the advantage of freeing the user from the burden of the prosthesis itself, overpassing several limitations regarding size, weight, or integration, and thus enables a much higher level of functionality.Approach.In this study, nine participants, two of whom suffer from severe upper-limb disabilities, teleoperated a humanoid assistive platform, performing complex bimanual tasks requiring high precision and bilateral arm/hand coordination, simulating home/office chores. A wearable body posture tracker was used for position control of the robotic torso and arms, while interactive machine learning applied to electromyography of the forearms helped the robot to build an increasingly accurate model of the participant's intent over time.Main results.All participants, irrespective of their disability, were uniformly able to perform the demanded tasks. Completion times, subjective evaluation scores, as well as energy- and time- efficiency show improvement over time on short and long term.Significance.This is the first time a hybrid setup, involving myoeletric and inertial measurements, is used by disabled people to teleoperate a bimanual humanoid robot. The proposed setup, taking advantage of interactive machine learning, is simple, non-invasive, and offers a new assistive solution for disabled people in their home environment. Additionnally, it has the potential of being used in several other applications in which fine humanoid robot control is required.
Collapse
Affiliation(s)
- Mathilde Connan
- Deutsches Zentrum fur Luft- und Raumfahrt Institut fur Robotik und Mechatronik, Muenchener Strasse 20, Oberpfaffenhofen-Wessling, 82234, GERMANY
| | - Marek Sierotowicz
- Deutsches Zentrum fur Luft- und Raumfahrt Institut fur Robotik und Mechatronik, Muenchener Strasse 20, Oberpfaffenhofen-Wessling, 82234, GERMANY
| | - Bernd Henze
- Deutsches Zentrum fur Luft- und Raumfahrt Institut fur Robotik und Mechatronik, Muenchener Strasse 20, Oberpfaffenhofen-Wessling, 82234, GERMANY
| | - Oliver Porges
- Deutsches Zentrum fur Luft- und Raumfahrt Institut fur Robotik und Mechatronik, Muenchener Strasse 20, Oberpfaffenhofen-Wessling, 82234, GERMANY
| | - Alin Albu-Schaeffer
- Deutsches Zentrum fur Luft- und Raumfahrt Institut fur Robotik und Mechatronik, Muenchener Strasse 20, Oberpfaffenhofen-Wessling, 82234, GERMANY
| | - Maximo Roa
- Deutsches Zentrum fur Luft- und Raumfahrt Institut fur Robotik und Mechatronik, Muenchener Strasse 20, Oberpfaffenhofen-Wessling, 82234, GERMANY
| | - Claudio Castellini
- Deutsches Zentrum fur Luft- und Raumfahrt DLR Institut fur Robotik und Mechatronik, Muenchener Strasse 20, Oberpfaffenhofen-Wessling, Bayern, 82234, GERMANY
| |
Collapse
|
10
|
McAnally K, Wallis G. Visual-haptic integration, action and embodiment in virtual reality. PSYCHOLOGICAL RESEARCH 2021; 86:1847-1857. [PMID: 34709463 DOI: 10.1007/s00426-021-01613-3] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 06/07/2021] [Accepted: 10/12/2021] [Indexed: 10/20/2022]
Abstract
The current generation of virtual reality (VR) technologies has improved substantially from legacy systems, particularly in the resolution and latency of their visual display. The presentation of haptic cues remains challenging, however, because haptic systems do not generalise well over the range of stimuli (both tactile and proprioceptive) normally present when interacting with objects in the world. This study investigated whether veridical tactile and proprioceptive cues lead to more efficient interaction with a virtual environment. Interaction in the world results in spatial and temporal correlation of tactile, proprioceptive and visual cues. When cues in VR are similarly correlated, observers experience a sense of embodiment and agency of their avatars. We investigated whether sensorimotor performance was mediated by embodiment of the avatar hands. Participants performed a Fitts' tapping task in different conditions (VR with no haptics, active haptics, passive haptics, and on a real touchscreen). The active-haptic condition provided abstract tactile cues and the passive haptic condition provided veridical tactile and proprioceptive cues. An additional (hybrid haptics) condition simulated an ideal passive haptic system. Movement efficiency (throughput) and embodiment were higher for the passive than for the active and no-haptics conditions. However, components of embodiment (perceived agency and ownership) did not predict unique variance in throughput. Improved sensorimotor performance and ratings of presence and realism support the use of passive haptics in VR environments where objects are in known and stable locations, regardless of whether performance was mediated by the sense of embodiment.
Collapse
Affiliation(s)
- Ken McAnally
- School of Human Movement and Nutrition Sciences, The University of Queensland, St. Lucia, QLD, 4072, Australia.
| | - Guy Wallis
- School of Human Movement and Nutrition Sciences, The University of Queensland, St. Lucia, QLD, 4072, Australia
| |
Collapse
|
11
|
Macchini M, De Matteis L, Schiano F, Floreano D. Personalized Human-Swarm Interaction Through Hand Motion. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3102324] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/08/2022]
|
12
|
Barresi G, Marinelli A, Caserta G, de Zambotti M, Tessadori J, Angioletti L, Boccardo N, Freddolini M, Mazzanti D, Deshpande N, Frigo CA, Balconi M, Gruppioni E, Laffranchi M, De Michieli L. Exploring the Embodiment of a Virtual Hand in a Spatially Augmented Respiratory Biofeedback Setting. Front Neurorobot 2021; 15:683653. [PMID: 34557082 PMCID: PMC8454775 DOI: 10.3389/fnbot.2021.683653] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 03/21/2021] [Accepted: 07/26/2021] [Indexed: 01/15/2023] Open
Abstract
Enhancing the embodiment of artificial limbs—the individuals' feeling that a virtual or robotic limb is integrated in their own body scheme—is an impactful strategy for improving prosthetic technology acceptance and human-machine interaction. Most studies so far focused on visuo-tactile strategies to empower the embodiment processes. However, novel approaches could emerge from self-regulation techniques able to change the psychophysiological conditions of an individual. Accordingly, this pilot study investigates the effects of a self-regulated breathing exercise on the processes of body ownership underlying the embodiment of a virtual right hand within a Spatially Augmented Respiratory Biofeedback (SARB) setting. This investigation also aims at evaluating the feasibility of the breathing exercise enabled by a low-cost SARB implementation designed for upcoming remote studies (a need emerged during the COVID-19 pandemic). Twenty-two subjects without impairments, and two transradial prosthesis users for a preparatory test, were asked (in each condition of a within-group design) to maintain a normal (about 14 breaths/min) or slow (about 6 breaths/min) respiratory rate to keep a static virtual right hand “visible” on a screen. Meanwhile, a computer-generated sphere moved from left to right toward the virtual hand during each trial (1 min) of 16. If the participant's breathing rate was within the target (slow or normal) range, a visuo-tactile event was triggered by the sphere passing under the virtual hand (the subjects observed it shaking while they perceived a vibratory feedback generated by a smartphone). Our results—mainly based on questionnaire scores and proprioceptive drift—highlight that the slow breathing condition induced higher embodiment than the normal one. This preliminary study reveals the feasibility and potential of a novel psychophysiological training strategy to enhance the embodiment of artificial limbs. Future studies are needed to further investigate mechanisms, efficacy and generalizability of the SARB techniques in training a bionic limb embodiment.
Collapse
Affiliation(s)
- Giacinto Barresi
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Andrea Marinelli
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy.,Department of Informatics, Bioengineering, Robotics, and Systems Engineering, Università degli Studi di Genova, Genoa, Italy
| | - Giulia Caserta
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy.,Movement Biomechanics and Motor Control Lab, Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| | | | - Jacopo Tessadori
- Visual Geometry and Modelling, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Laura Angioletti
- International Research Center for Cognitive Applied Neuroscience, Università Cattolica del Sacro Cuore, Milan, Italy.,Research Unit in Affective and Social Neuroscience, Department of Psychology, Università Cattolica del Sacro Cuore, Milan, Italy
| | - Nicolò Boccardo
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Marco Freddolini
- Rehab Technologies, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Dario Mazzanti
- Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Nikhil Deshpande
- Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy
| | - Carlo Albino Frigo
- Movement Biomechanics and Motor Control Lab, Department of Electronics, Information and Bioengineering, Politecnico di Milano, Milan, Italy
| | - Michela Balconi
- International Research Center for Cognitive Applied Neuroscience, Università Cattolica del Sacro Cuore, Milan, Italy.,Research Unit in Affective and Social Neuroscience, Department of Psychology, Università Cattolica del Sacro Cuore, Milan, Italy
| | - Emanuele Gruppioni
- Centro Protesi INAIL, Istituto Nazionale per l'Assicurazione contro gli Infortuni sul Lavoro, Bologna, Italy
| | | | | |
Collapse
|
13
|
Rosa N, Veltkamp RC, Hürst W, Brouwer AM, Gijsbertse K, Cocu I, Werkhoven P. Embodiment and Performance in the Supernumerary Hand Illusion in Augmented Reality. FRONTIERS IN COMPUTER SCIENCE 2021. [DOI: 10.3389/fcomp.2021.694916] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/13/2022] Open
Abstract
In teleoperations, robots are generally used because related tasks are too dangerous, uncomfortable or impossible for humans to perform. When using augmented reality to control robotic limbs in such teleoperations, it is essential to understand how these extra virtual limbs are experienced. In particular, the relationship between the embodiment experience of the user and relevant outcomes such as task performance must be examined. In this article, we study the relationship between experienced embodiment of a supernumerary virtual arm that acts alongside a user’s two real arms, and their task performance in augmented reality. Specifically, we compare how well users can trace a virtual half ring placed just outside of personal space using their virtual arm in a condition where there is expected to be low embodiment (a floating disconnected hand) and a condition where there is expected to be high embodiment (a connected arm and hand). Embodiment is measured quantitatively through skin conductance response and qualitatively through ownership, agency, and self-location questionnaires. Performance is measured in terms of tracing precision. The results show positive correlations between subjective ownership and agency, and agency and performance, but no correlation between subjective or objective ownership and performance. Also, ownership ratings were low overall, while the agency ratings were significantly higher for the disconnected hand condition than the connected arm condition, as was performance. Notably, the presence of the virtual arm evoked incorrect expectations of the movement capabilities of the arm, which may have contributed to an overall preference for the unrealistic disconnected hand over the more realistic connected arm in this particular task. Our results imply that methods to increase performance in various teleoperations can indeed be found in the experience of embodiment: not necessarily directly through ownership, but through ownership mediated by agency.
Collapse
|
14
|
Chiappe D, Vervaeke J. Distributed Cognition and the Experience of Presence in the Mars Exploration Rover Mission. Front Psychol 2021; 12:689932. [PMID: 34234725 PMCID: PMC8255671 DOI: 10.3389/fpsyg.2021.689932] [Citation(s) in RCA: 1] [Impact Index Per Article: 0.3] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 04/01/2021] [Accepted: 05/24/2021] [Indexed: 12/03/2022] Open
Abstract
Although research on presence in virtual environments has increased in the last few decades due to the rise of immersive technologies, it has not examined how it is achieved in distributed cognitive systems. To this end, we examine the sense of presence on the Martian landscape experienced by scientific team members in the Mars Exploration Rover (MER) mission (2004–2018). How this was achieved is not obvious because the sensorimotor coupling that typically underlies presence in mundane situations was absent. Nonetheless, we argue that the Three-Level model can provide a framework for exploring how presence was achieved. This account distinguishes between proto-presence, core-presence, and extended-presence, each level dependent on being able to respond effectively to affordances at a particular level of abstraction, operating at different timescales. We maintain that scientists' sense of presence on Mars involved core-presence and extended-presence rather than proto-presence. Extended-presence involved successfully establishing distal intentions (D-intentions) during strategic planning, i.e., long term conceptual goals. Core-presence involved successfully enacting proximal intentions (P-intentions) during tactical planning by carrying out specific actions on a particular target, abstracting away from sensorimotor details. This was made possible by team members “becoming the rover,” which enhanced their ability to identify relevant affordances revealed through images. We argue, however, that because Mars exploration is a collective activity involving shared agency by a distributed cognitive system, the experience of presence was a collective presence of the team through the rover.
Collapse
Affiliation(s)
- Dan Chiappe
- Department of Psychology, California State University, Long Beach, CA, United States
| | - John Vervaeke
- Department of Psychology, University of Toronto, Toronto, ON, Canada
| |
Collapse
|
15
|
Shin M, Lee S, Song SW, Chung D. Enhancement of perceived body ownership in virtual reality-based teleoperation may backfire in the execution of high-risk tasks. COMPUTERS IN HUMAN BEHAVIOR 2021; 115:106605. [PMID: 33100505 PMCID: PMC7568672 DOI: 10.1016/j.chb.2020.106605] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/09/2020] [Revised: 10/07/2020] [Accepted: 10/14/2020] [Indexed: 11/26/2022]
Abstract
Guided by previous research on the role of embodiment in virtual environments, this study aimed to investigate the potential effects of using human-like (compared to robotic) virtual hands on work performances in the context of virtual reality (VR)-based teleoperation of high-risk machinery. A 2 × 2 mixed factorial design experiment (N = 74), with the virtual hand representation as a within-subjects factor (robotic vs. human-like virtual hands) and the risk of danger as a between-subjects factor (low vs. high), was conducted to examine the effects of virtual hand representations (i.e., human-likeness) on perceived body ownership (i.e., embodiment), risk perception, intention to work using the teleoperator, and work performance (i.e., the number of successful task completions). In addition, the moderating effects of the risk of danger on the relationship between perceived body ownership and risk perception were explored. Results showed that the enhancement of perceived body ownership in VR-based teleoperation, induced by the use of human-like hands, increased the risk perception and degraded workers' task performances in the execution of high-risk tasks. Further implications of the findings were discussed.
Collapse
Affiliation(s)
- Mincheol Shin
- Department of Communication and Cognition, Tilburg University, Tilburg, Netherlands
| | - Sanguk Lee
- Department of Communication, Michigan State University, MI, USA
| | - Stephen W Song
- Comm. & Tech. Lab, Kwangwoon University, Seoul, South Korea
| | - Donghun Chung
- School of Media and Communication, Kwangwoon University, Seoul, South Korea
| |
Collapse
|
16
|
Nostadt N, Abbink DA, Christ O, Beckerle P. Embodiment, Presence, and Their Intersections. ACM TRANSACTIONS ON HUMAN-ROBOT INTERACTION 2020. [DOI: 10.1145/3389210] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.5] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 10/24/2022]
Abstract
Subjective experience of human control over remote, artificial, or virtual limbs has traditionally been investigated from two separate angles: presence research originates from teleoperation, aiming to capture to what extent the user feels like actually being in the remote or virtual environment. Embodiment captures to what extent a virtual or artificial limb is perceived as one’s own limb. Unfortunately, the two research fields have not interacted much. This survey intends to provide a coherent overview of the literature at the intersection of these two fields to further that interaction. Two rounds of systematic research in topic-related data bases resulted in 414 related articles, 14 of which satisfy the deliberately strict inclusion criteria: 2 theoretical frameworks that highlighted intersections and 12 experimental studies that evaluated subjective measures for both concepts. Considering the surrounding literature as well, theoretical and experimental potential of embodiment and presence are discussed and suggestions to apply them in teleoperation research are derived. While increased publication activity is observed between 2016 and 2018, potentially caused by affordable virtual reality technologies, various open questions remain. To tackle them, human-in-the-loop experiments and three guiding principles for teleoperation system design (mechanical fidelity, spatial bodily awareness, and self-identification) are suggested.
Collapse
Affiliation(s)
| | - David A. Abbink
- Delft Haptics Lab, Department of Cognitive Robotics, Faculty 3mE, Delft University of Technology, The Netherlands
| | - Oliver Christ
- Institute Humans in Complex Systems, School of Applied Psychology, University of Applied Sciences and Arts Northwestern Switzerland, Switzerland
| | - Philipp Beckerle
- Elastic Lightweight Robotics Group, Robotics Research Institute, Technische Universität Dortmund, German and Institute for Mechatronic Systems in Mechanical Engineering, Technische Universität Darmstadt, Darmstadt, Germany
| |
Collapse
|