1
|
Dominijanni G, Pinheiro DL, Pollina L, Orset B, Gini M, Anselmino E, Pierella C, Olivier J, Shokur S, Micera S. Human motor augmentation with an extra robotic arm without functional interference. Sci Robot 2023; 8:eadh1438. [PMID: 38091424 DOI: 10.1126/scirobotics.adh1438] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/13/2023] [Accepted: 11/15/2023] [Indexed: 12/18/2023]
Abstract
Extra robotic arms (XRAs) are gaining interest in neuroscience and robotics, offering potential tools for daily activities. However, this compelling opportunity poses new challenges for sensorimotor control strategies and human-machine interfaces (HMIs). A key unsolved challenge is allowing users to proficiently control XRAs without hindering their existing functions. To address this, we propose a pipeline to identify suitable HMIs given a defined task to accomplish with the XRA. Following such a scheme, we assessed a multimodal motor HMI based on gaze detection and diaphragmatic respiration in a purposely designed modular neurorobotic platform integrating virtual reality and a bilateral upper limb exoskeleton. Our results show that the proposed HMI does not interfere with speaking or visual exploration and that it can be used to control an extra virtual arm independently from the biological ones or in coordination with them. Participants showed significant improvements in performance with daily training and retention of learning, with no further improvements when artificial haptic feedback was provided. As a final proof of concept, naïve and experienced participants used a simplified version of the HMI to control a wearable XRA. Our analysis indicates how the presented HMI can be effectively used to control XRAs. The observation that experienced users achieved a success rate 22.2% higher than that of naïve users, combined with the result that naïve users showed average success rates of 74% when they first engaged with the system, endorses the viability of both the virtual reality-based testing and training and the proposed pipeline.
Collapse
Affiliation(s)
- Giulia Dominijanni
- Neuro-X Institute, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Daniel Leal Pinheiro
- Neuro-X Institute, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
- Neuroengineering and Neurocognition Laboratory, Escola Paulista de Medicina, Department of Neurology and Neurosurgery, Division of Neuroscience, Universidade Federal de São Paulo, São Paulo, Brazil
| | - Leonardo Pollina
- Neuro-X Institute, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Bastien Orset
- Neuro-X Institute, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
| | - Martina Gini
- BioRobotics Institute, Health Interdisciplinary Center, and Department of Excellence in AI and Robotics, Scuola Superiore Sant'Anna, Pisa, Italy
- Neuroelectronic Interfaces, Faculty of Electrical Engineering and IT, Rheinisch-Westfälische Technische Hochschule (RWTH) Aachen, Aachen 52074, Germany
| | - Eugenio Anselmino
- BioRobotics Institute, Health Interdisciplinary Center, and Department of Excellence in AI and Robotics, Scuola Superiore Sant'Anna, Pisa, Italy
| | - Camilla Pierella
- Department of Neurosciences, Rehabilitation, Ophthalmology, Genetics, and Maternal and Children's Sciences (DINOGMI), University of Genoa, Genoa, Italy
| | - Jérémy Olivier
- Institute for Industrial Sciences and Technologies, Haute Ecole du Paysage, d'Ingénierie et d'Architecture (HEPIA), HES-SO University of Applied Sciences and Arts Western Switzerland, Geneva, Switzerland
| | - Solaiman Shokur
- Neuro-X Institute, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
- BioRobotics Institute, Health Interdisciplinary Center, and Department of Excellence in AI and Robotics, Scuola Superiore Sant'Anna, Pisa, Italy
| | - Silvestro Micera
- Neuro-X Institute, Ecole Polytechnique Fédérale de Lausanne, Lausanne, Switzerland
- BioRobotics Institute, Health Interdisciplinary Center, and Department of Excellence in AI and Robotics, Scuola Superiore Sant'Anna, Pisa, Italy
| |
Collapse
|
2
|
Sanmartin-Senent A, Pena-Perez N, Burdet E, Eden J. Redundancy Resolution in Trimanual vs. Bimanual Tracking Tasks. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2023; 2023:1-5. [PMID: 38083745 DOI: 10.1109/embc40787.2023.10340722] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 12/18/2023]
Abstract
Supernumerary limbs promise to allow users to perform complex tasks that would otherwise require the actions of teams. However, how the user's capability for multimanual coordination compares to bimanual coordination, and how the motor system decides to configure its limb contributions given task redundancy is unclear. We conducted bimanual and trimanual (with the foot as a third-hand controller) virtual reality visuomotor tracking experiments to study how 32 healthy participants changed their limb coordination in response to uninstructed cursor mapping changes. This used a shared cursor mapped to the average limbs' position for different limb combinations. The results show that most participants correctly identified the different mappings during bimanual tracking, and accordingly minimized task-irrelevant motion. Instead during trimanual coordination, participants consistently moved all three limbs concurrently, showing weaker ipsilateral hand-foot coordination. These findings show how redundancy resolution and the resulting coordination patterns differ between similar bimanual and trimanual tasks. Further research is needed to consider the effect of learning on coordination behaviour.
Collapse
|
3
|
Pinardi M, Noccaro A, Raiano L, Formica D, Di Pino G. Comparing end-effector position and joint angle feedback for online robotic limb tracking. PLoS One 2023; 18:e0286566. [PMID: 37289675 PMCID: PMC10249844 DOI: 10.1371/journal.pone.0286566] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/04/2022] [Accepted: 05/18/2023] [Indexed: 06/10/2023] Open
Abstract
Somatosensation greatly increases the ability to control our natural body. This suggests that supplementing vision with haptic sensory feedback would also be helpful when a user aims at controlling a robotic arm proficiently. However, whether the position of the robot and its continuous update should be coded in a extrinsic or intrinsic reference frame is not known. Here we compared two different supplementary feedback contents concerning the status of a robotic limb in 2-DoFs configuration: one encoding the Cartesian coordinates of the end-effector of the robotic arm (i.e., Task-space feedback) and another and encoding the robot joints angles (i.e., Joint-space feedback). Feedback was delivered to blindfolded participants through vibrotactile stimulation applied on participants' leg. After a 1.5-hour training with both feedbacks, participants were significantly more accurate with Task compared to Joint-space feedback, as shown by lower position and aiming errors, albeit not faster (i.e., similar onset delay). However, learning index during training was significantly higher in Joint space feedback compared to Task-space feedback. These results suggest that Task-space feedback is probably more intuitive and more suited for activities which require short training sessions, while Joint space feedback showed potential for long-term improvement. We speculate that the latter, despite performing worse in the present work, might be ultimately more suited for applications requiring long training, such as the control of supernumerary robotic limbs for surgical robotics, heavy industrial manufacturing, or more generally, in the context of human movement augmentation.
Collapse
Affiliation(s)
- Mattia Pinardi
- NEXT: Neurophysiology and Neuroengineering of Human-Technology Interaction Research Unit, Università Campus Bio-Medico di Roma, Rome, Italy
| | - Alessia Noccaro
- Neurorobotics Group, Newcastle University, Newcastle, United Kingdom
| | - Luigi Raiano
- NEXT: Neurophysiology and Neuroengineering of Human-Technology Interaction Research Unit, Università Campus Bio-Medico di Roma, Rome, Italy
| | - Domenico Formica
- Neurorobotics Group, Newcastle University, Newcastle, United Kingdom
| | - Giovanni Di Pino
- NEXT: Neurophysiology and Neuroengineering of Human-Technology Interaction Research Unit, Università Campus Bio-Medico di Roma, Rome, Italy
| |
Collapse
|
4
|
Genay A, Lecuyer A, Hachet M. Being an Avatar "for Real": A Survey on Virtual Embodiment in Augmented Reality. IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS 2022; 28:5071-5090. [PMID: 34310309 DOI: 10.1109/tvcg.2021.3099290] [Citation(s) in RCA: 6] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/13/2023]
Abstract
Virtual self-avatars have been increasingly used in Augmented Reality (AR) where one can see virtual content embedded into physical space. However, little is known about the perception of self-avatars in such a context. The possibility that their embodiment could be achieved in a similar way as in Virtual Reality opens the door to numerous applications in education, communication, entertainment, or the medical field. This article aims to review the literature covering the embodiment of virtual self-avatars in AR. Our goal is (i) to guide readers through the different options and challenges linked to the implementation of AR embodiment systems, (ii) to provide a better understanding of AR embodiment perception by classifying the existing knowledge, and (iii) to offer insight on future research topics and trends for AR and avatar research. To do so, we introduce a taxonomy of virtual embodiment experiences by defining a "body avatarization" continuum. The presented knowledge suggests that the sense of embodiment evolves in the same way in AR as in other settings, but this possibility has yet to be fully investigated. We suggest that, whilst it is yet to be well understood, the embodiment of avatars has a promising future in AR and conclude by discussing possible directions for research.
Collapse
|
5
|
Eden J, Bräcklein M, Ibáñez J, Barsakcioglu DY, Di Pino G, Farina D, Burdet E, Mehring C. Principles of human movement augmentation and the challenges in making it a reality. Nat Commun 2022; 13:1345. [PMID: 35292665 PMCID: PMC8924218 DOI: 10.1038/s41467-022-28725-7] [Citation(s) in RCA: 19] [Impact Index Per Article: 9.5] [Reference Citation Analysis] [Abstract] [Key Words] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 05/29/2021] [Accepted: 02/04/2022] [Indexed: 12/23/2022] Open
Abstract
Augmenting the body with artificial limbs controlled concurrently to one's natural limbs has long appeared in science fiction, but recent technological and neuroscientific advances have begun to make this possible. By allowing individuals to achieve otherwise impossible actions, movement augmentation could revolutionize medical and industrial applications and profoundly change the way humans interact with the environment. Here, we construct a movement augmentation taxonomy through what is augmented and how it is achieved. With this framework, we analyze augmentation that extends the number of degrees-of-freedom, discuss critical features of effective augmentation such as physiological control signals, sensory feedback and learning as well as application scenarios, and propose a vision for the field.
Collapse
Affiliation(s)
- Jonathan Eden
- Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, UK
| | - Mario Bräcklein
- Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, UK
| | - Jaime Ibáñez
- Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, UK
- BSICoS, IIS Aragón, Universidad de Zaragoza, Zaragoza, Spain
- Department of Clinical and Movement Neurosciences, Institute of Neurology, University College London, London, UK
| | | | - Giovanni Di Pino
- NEXT: Neurophysiology and Neuroengineering of Human-Technology Interaction Research Unit, Università Campus Bio-Medico di Roma, Rome, Italy
| | - Dario Farina
- Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, UK
| | - Etienne Burdet
- Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, UK.
| | - Carsten Mehring
- Bernstein Center Freiburg, University of Freiburg, Freiburg im Breisgau, 79104, Germany
- Faculty of Biology, University of Freiburg, Freiburg im Breisgau, 79104, Germany
| |
Collapse
|
6
|
Prieto Prada JD, Im J, Oh H, Song C. Enhanced location tracking in sensor fusion-assisted virtual reality micro-manipulation environments. PLoS One 2021; 16:e0261933. [PMID: 34962969 PMCID: PMC8714085 DOI: 10.1371/journal.pone.0261933] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 09/15/2020] [Accepted: 12/14/2021] [Indexed: 12/02/2022] Open
Abstract
Virtual reality (VR) technology plays a significant role in many biomedical applications. These VR scenarios increase the valuable experience of tasks requiring great accuracy with human subjects. Unfortunately, commercial VR controllers have large positioning errors in a micro-manipulation task. Here, we propose a VR-based framework along with a sensor fusion algorithm to improve the microposition tracking performance of a microsurgical tool. To the best of our knowledge, this is the first application of Kalman filter in a millimeter scale VR environment, by using the position data between the VR controller and an inertial measuring device. This study builds and tests two cases: (1) without sensor fusion tracking and (2) location tracking with active sensor fusion. The static and dynamic experiments demonstrate that the Kalman filter can provide greater precision during micro-manipulation in small scale VR scenarios.
Collapse
Affiliation(s)
| | - Jintaek Im
- Department of Robotics Engineering, DGIST, Daegu, South Korea
| | - Hyondong Oh
- School of Mechanical, Aerospace and Nuclear Engineering, UNIST, Ulsan, South Korea
| | - Cheol Song
- Department of Robotics Engineering, DGIST, Daegu, South Korea
- * E-mail:
| |
Collapse
|
7
|
Shafti A, Haar S, Mio R, Guilleminot P, Faisal AA. Playing the piano with a robotic third thumb: assessing constraints of human augmentation. Sci Rep 2021; 11:21375. [PMID: 34725355 PMCID: PMC8560761 DOI: 10.1038/s41598-021-00376-6] [Citation(s) in RCA: 3] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/22/2021] [Accepted: 10/05/2021] [Indexed: 11/16/2022] Open
Abstract
Contemporary robotics gives us mechatronic capabilities for augmenting human bodies with extra limbs. However, how our motor control capabilities pose limits on such augmentation is an open question. We developed a Supernumerary Robotic 3rd Thumbs (SR3T) with two degrees-of-freedom controlled by the user’s body to endow them with an extra contralateral thumb on the hand. We demonstrate that a pianist can learn to play the piano with 11 fingers within an hour. We then evaluate 6 naïve and 6 experienced piano players in their prior motor coordination and their capability in piano playing with the robotic augmentation. We show that individuals’ augmented performance with the SR3T could be explained by our new custom motor coordination assessment, the Human Augmentation Motor Coordination Assessment (HAMCA) performed pre-augmentation. Our work demonstrates how supernumerary robotics can augment humans in skilled tasks and that individual differences in their augmentation capability are explainable by their individual motor coordination abilities.
Collapse
Affiliation(s)
- Ali Shafti
- Brain and Behaviour Laboratory, Department of Bioengineering, Imperial College London, London, SW7 2AZ, UK.,Department of Computing, Imperial College London, London, SW7 2AZ, UK.,Behaviour Analytics Laboratory, Data Science Institute, London, SW7 2AZ, UK
| | - Shlomi Haar
- Brain and Behaviour Laboratory, Department of Bioengineering, Imperial College London, London, SW7 2AZ, UK.,Behaviour Analytics Laboratory, Data Science Institute, London, SW7 2AZ, UK.,Department of Brain Sciences and UK Dementia Research Institute - Care Research and Technology Centre, Imperial College London, London, W12 0BZ, UK
| | - Renato Mio
- Brain and Behaviour Laboratory, Department of Bioengineering, Imperial College London, London, SW7 2AZ, UK
| | - Pierre Guilleminot
- Brain and Behaviour Laboratory, Department of Bioengineering, Imperial College London, London, SW7 2AZ, UK
| | - A Aldo Faisal
- Brain and Behaviour Laboratory, Department of Bioengineering, Imperial College London, London, SW7 2AZ, UK. .,Department of Computing, Imperial College London, London, SW7 2AZ, UK. .,Behaviour Analytics Laboratory, Data Science Institute, London, SW7 2AZ, UK. .,UKRI CDT in AI for Healthcare, Imperial College London, London, SW7 2AZ, UK. .,MRC London Institute of Medical Sciences, London, W12 0NN, UK.
| |
Collapse
|
8
|
Dominijanni G, Shokur S, Salvietti G, Buehler S, Palmerini E, Rossi S, De Vignemont F, d’Avella A, Makin TR, Prattichizzo D, Micera S. The neural resource allocation problem when enhancing human bodies with extra robotic limbs. NAT MACH INTELL 2021. [DOI: 10.1038/s42256-021-00398-9] [Citation(s) in RCA: 9] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 01/19/2023]
|
9
|
Noccaro A, Eden J, Di Pino G, Formica D, Burdet E. Human performance in three-hands tasks. Sci Rep 2021; 11:9511. [PMID: 33947906 PMCID: PMC8096970 DOI: 10.1038/s41598-021-88862-9] [Citation(s) in RCA: 10] [Impact Index Per Article: 3.3] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 02/05/2021] [Accepted: 04/19/2021] [Indexed: 11/09/2022] Open
Abstract
The successful completion of complex tasks like hanging a picture or laparoscopic surgery requires coordinated motion of more than two limbs. User-controlled supernumerary robotic limbs (SL) have been proposed to bypass the need for coordination with a partner in such tasks. However, neither the capability to control multiple limbs alone relative to collaborative control with partners, nor how that capability varies across different tasks, is well understood. In this work, we present an investigation of tasks requiring three-hands where the foot was used as an additional source of motor commands. We considered: (1) how does simultaneous control of three hands compare to a cooperating dyad; (2) how this relative performance was altered by the existence of constraints emanating from real or virtual physical connections (mechanical constraints) or from cognitive limits (cognitive constraints). It was found that a cooperating dyad outperformed a single user in all scenarios in terms of task score, path efficiency and motion smoothness. However, while the participants were able to reach more targets with increasing mechanical constraints/decreasing number of simultaneous goals, the relative difference in performance between a dyad and a participant performing trimanual activities decreased, suggesting further potential for SLs in this class of scenario.
Collapse
Affiliation(s)
- A Noccaro
- NEXT: Neurophysiology and Neuroengineering of Human-Technology Interaction Research Unit, Università Campus Bio-Medico di Roma, Rome, Italy.
| | - J Eden
- Department of Bioengineering, Imperial College of Science Technology and Medicine, London, UK
| | - G Di Pino
- NEXT: Neurophysiology and Neuroengineering of Human-Technology Interaction Research Unit, Università Campus Bio-Medico di Roma, Rome, Italy
| | - D Formica
- NEXT: Neurophysiology and Neuroengineering of Human-Technology Interaction Research Unit, Università Campus Bio-Medico di Roma, Rome, Italy
| | - E Burdet
- Department of Bioengineering, Imperial College of Science Technology and Medicine, London, UK
| |
Collapse
|
10
|
Song H, Asada HH. Integrated Voluntary-Reactive Control of a Human-SuperLimb Hybrid System for Hemiplegic Patient Support. IEEE Robot Autom Lett 2021. [DOI: 10.1109/lra.2021.3058926] [Citation(s) in RCA: 2] [Impact Index Per Article: 0.7] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/07/2022]
|
11
|
Palee P, Wongta N, Khwanngern K, Jitmun W, Choosri N. Serious Game for Teaching Undergraduate Medical Students in Cleft lip and Palate Treatment Protocol. Int J Med Inform 2020; 141:104166. [PMID: 32570197 DOI: 10.1016/j.ijmedinf.2020.104166] [Citation(s) in RCA: 4] [Impact Index Per Article: 1.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/05/2019] [Revised: 03/30/2020] [Accepted: 04/30/2020] [Indexed: 12/31/2022]
Abstract
OBJECTIVES To develop and evaluate a serious game to deliver knowledge about the cleft lip with or without cleft palate (CL/P) protocol used in Craniofacial Center, Chiang Mai University. METHODS The game "Cleft Island" was developed then extensively evaluated using different criteria, including the measured knowledge gained by participants, and gameplay experience. A group of 53 fourth and fifth-year medical students (male: 24, female: 29) were recruited to take part in an experiment. They were from the Faculty of Medicine, Chiang Mai University, and had just rotated to the Department of Surgery. Three evaluations were conducted: 1) to test whether the players had competently gained CL/P knowledge; 2) to evaluate the usability of the game according to the System Usability Scale (SUS); and 3) to evaluate the game experience in terms of a Game Experience Questionnaire (GEQ) test. RESULTS The results indicate a statistically significant improvement of medical students' knowledge after performing the Wilcoxon Signed-Ranks test (p < 0.5) between pre-test and post-test scores of the same medical student group; the acceptable average SUS score (M = 55.28) of the serious game; a moderate degree of experience of the GEQ components including positive affect (M = 2.64), competence (M = 2.49), and immersion (M = 2.21). CONCLUSIONS Cleft Island can be used as an effective supplementary instructional material, which has the potential to provide significant knowledge of CL/P treatment protocol for the players. As far as the authors are aware, this is the first study to implement and assess a serious game for training in CL/P protocol.
Collapse
Affiliation(s)
- Patison Palee
- College of Arts, Media, and Technology, Chiang Mai University, 239 Huaykaew Rd., Suthep, Muang, Chiang Mai, 50200, Thailand
| | - Noppon Wongta
- College of Arts, Media, and Technology, Chiang Mai University, 239 Huaykaew Rd., Suthep, Muang, Chiang Mai, 50200, Thailand
| | - Krit Khwanngern
- Faculty of Medicine, Chiang Mai University, 110 Inthawarorot Rd., Suthep, Muang, Chiang Mai, 50200, Thailand; Center of Data Analytics and Knowledge Synthesis for Healthcare, 239 Huaykaew Rd., Suthep, Muang, Chiang Mai, 50200, Thailand
| | - Waritsara Jitmun
- College of Arts, Media, and Technology, Chiang Mai University, 239 Huaykaew Rd., Suthep, Muang, Chiang Mai, 50200, Thailand
| | - Noppon Choosri
- College of Arts, Media, and Technology, Chiang Mai University, 239 Huaykaew Rd., Suthep, Muang, Chiang Mai, 50200, Thailand; Center of Data Analytics and Knowledge Synthesis for Healthcare, 239 Huaykaew Rd., Suthep, Muang, Chiang Mai, 50200, Thailand.
| |
Collapse
|
12
|
Huang Y, Burdet E, Cao L, Phan PT, Tiong AMH, Zheng P, Phee SJ. Performance Evaluation of a Foot Interface to Operate a Robot Arm. IEEE Robot Autom Lett 2019. [DOI: 10.1109/lra.2019.2926215] [Citation(s) in RCA: 11] [Impact Index Per Article: 2.2] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
13
|
Bashford L, Wu J, Sarma D, Collins K, Rao RPN, Ojemann JG, Mehring C. Concurrent control of a brain-computer interface and natural overt movements. J Neural Eng 2018; 15:066021. [PMID: 30303130 DOI: 10.1088/1741-2552/aadf3d] [Citation(s) in RCA: 7] [Impact Index Per Article: 1.2] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 12/21/2022]
Abstract
OBJECTIVE A primary control signal in brain-computer interfaces (BCIs) have been cortical signals related to movement. However, in cases where natural motor function remains, BCI control signals may interfere with other possibly simultaneous activity for useful ongoing movement. We sought to determine if the brain could learn to control both a BCI and concurrent overt movement execution in such cases. APPROACH We designed experiments where BCI and overt movements must be used concurrently and in coordination to achieve a 2D centre out control. Power in the 70-90 Hz band of human electrocorticography (ECoG) signals, was used to generate BCI control commands for vertical movement of the cursor. These signals were deliberately recorded from the same human cortical site that produced the strongest movement related activity associated with the concurrent overt finger movements required for the horizontal movement of the cursor. MAIN RESULTS We demonstrate that three subjects were able to perform the concurrent BCI task, controlling BCI and natural movements simultaneously and to a large extent independently. We conclude that the brain is capable of dissociating the original control signal dependency on movement, producing specific BCI control signals in the presence of motor related responses from the ongoing overt behaviour with which the BCI signal was initially correlated. SIGNIFICANCE We demonstrate a novel human brain-computer interface (BCI) which can be used to control movement concurrently and in coordination with movements of the natural limbs. This demonstrates the dissociation of cortical activity from the behaviour with which it was originally associated despite the ongoing behaviour and shows the feasibility of achieving simultaneous BCI control of devices with natural movements.
Collapse
Affiliation(s)
- L Bashford
- Department of Bioengineering, Imperial College London, London, United Kingdom. Bernstein Center and Brain-Links Brain-Tools, University of Freiburg, Freiburg, Germany
| | | | | | | | | | | | | |
Collapse
|
14
|
Hussain I, Santarnecchi E, Leo A, Ricciardi E, Rossi S, Prattichizzo D. A magnetic compatible supernumerary robotic finger for functional magnetic resonance imaging (fMRI) acquisitions: Device description and preliminary results. IEEE Int Conf Rehabil Robot 2017; 2017:1177-1182. [PMID: 28813981 DOI: 10.1109/icorr.2017.8009409] [Citation(s) in RCA: 4] [Impact Index Per Article: 0.6] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/10/2022]
Abstract
The Supernumerary robotic limbs are a recently introduced class of wearable robots that, differently from traditional prostheses and exoskeletons, aim at adding extra effectors (i.e., arms, legs, or fingers) to the human user, rather than substituting or enhancing the natural ones. However, it is still undefined whether the use of supernumerary robotic limbs could specifically lead to neural modifications in brain dynamics. The illusion of owning the part of body has been already proven in many experimental observations, such as those relying on multisensory integration (e.g., rubber hand illusion), prosthesis and even on virtual reality. In this paper we present a description of a novel magnetic compatible supernumerary robotic finger together with preliminary observations from two functional magnetic resonance imaging (fMRI) experiments, in which brain activity was measured before and after a period of training with the robotic device, and during the use of the novel MRI-compatible version of the supernumerary robotic finger. Results showed that the usage of the MR-compatible robotic finger is safe and does not produce artifacts on MRI images. Moreover, the training with the supernumerary robotic finger recruits a network of motor-related cortical regions (i.e. primary and supplementary motor areas), hence the same motor network of a fully physiological voluntary motor gestures.
Collapse
|
15
|
Abdi E, Bouri M, Burdet E, Himidan S, Bleuler H. Positioning the endoscope in laparoscopic surgery by foot: Influential factors on surgeons' performance in virtual trainer. ANNUAL INTERNATIONAL CONFERENCE OF THE IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. IEEE ENGINEERING IN MEDICINE AND BIOLOGY SOCIETY. ANNUAL INTERNATIONAL CONFERENCE 2017; 2017:3944-3948. [PMID: 29060760 DOI: 10.1109/embc.2017.8037719] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/07/2023]
Abstract
We have investigated how surgeons can use the foot to position a laparoscopic endoscope, a task that normally requires an extra assistant. Surgeons need to train in order to exploit the possibilities offered by this new technique and safely manipulate the endoscope together with the hands movements. A realistic abdominal cavity has been developed as training simulator to investigate this multi-arm manipulation. In this virtual environment, the surgeon's biological hands are modelled as laparoscopic graspers while the viewpoint is controlled by the dominant foot. 23 surgeons and medical students performed single-handed and bimanual manipulation in this environment. The results show that residents had superior performance compared to both medical students and more experienced surgeons, suggesting that residency is an ideal period for this training. Performing the single-handed task improves the performance in the bimanual task, whereas the converse was not true.
Collapse
|
16
|
Bashford L, Mehring C. Ownership and Agency of an Independent Supernumerary Hand Induced by an Imitation Brain-Computer Interface. PLoS One 2016; 11:e0156591. [PMID: 27303808 PMCID: PMC4909224 DOI: 10.1371/journal.pone.0156591] [Citation(s) in RCA: 9] [Impact Index Per Article: 1.1] [Reference Citation Analysis] [Abstract] [MESH Headings] [Grants] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/17/2015] [Accepted: 05/17/2016] [Indexed: 11/19/2022] Open
Abstract
To study body ownership and control, illusions that elicit these feelings in non-body objects are widely used. Classically introduced with the Rubber Hand Illusion, these illusions have been replicated more recently in virtual reality and by using brain-computer interfaces. Traditionally these illusions investigate the replacement of a body part by an artificial counterpart, however as brain-computer interface research develops it offers us the possibility to explore the case where non-body objects are controlled in addition to movements of our own limbs. Therefore we propose a new illusion designed to test the feeling of ownership and control of an independent supernumerary hand. Subjects are under the impression they control a virtual reality hand via a brain-computer interface, but in reality there is no causal connection between brain activity and virtual hand movement but correct movements are observed with 80% probability. These imitation brain-computer interface trials are interspersed with movements in both the subjects' real hands, which are in view throughout the experiment. We show that subjects develop strong feelings of ownership and control over the third hand, despite only receiving visual feedback with no causal link to the actual brain signals. Our illusion is crucially different from previously reported studies as we demonstrate independent ownership and control of the third hand without loss of ownership in the real hands.
Collapse
Affiliation(s)
- Luke Bashford
- Department of Bioengineering, Imperial College London, London, United Kingdom
- Bernstein Centre Freiburg, University of Freiburg, Freiburg, Germany
- * E-mail:
| | - Carsten Mehring
- Bernstein Centre Freiburg, University of Freiburg, Freiburg, Germany
| |
Collapse
|
17
|
Abdi E, Burdet E, Bouri M, Himidan S, Bleuler H. In a demanding task, three-handed manipulation is preferred to two-handed manipulation. Sci Rep 2016; 6:21758. [PMID: 26912293 PMCID: PMC4766403 DOI: 10.1038/srep21758] [Citation(s) in RCA: 19] [Impact Index Per Article: 2.4] [Reference Citation Analysis] [Abstract] [MESH Headings] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/24/2015] [Accepted: 01/28/2016] [Indexed: 11/08/2022] Open
Abstract
Equipped with a third hand under their direct control, surgeons may be able to perform certain surgical interventions alone; this would reduce the need for a human assistant and related coordination difficulties. However, does human performance improve with three hands compared to two hands? To evaluate this possibility, we carried out a behavioural study on the performance of naive adults catching objects with three virtual hands controlled by their two hands and right foot. The subjects could successfully control the virtual hands in a few trials. With this control strategy, the workspace of the hands was inversely correlated with the task velocity. The comparison of performance between the three and two hands control revealed no significant difference of success in catching falling objects and in average effort during the tasks. Subjects preferred the three handed control strategy, found it easier, with less physical and mental burden. Although the coordination of the foot with the natural hands increased trial after trial, about two minutes of practice was not sufficient to develop a sense of ownership towards the third arm.
Collapse
Affiliation(s)
- Elahe Abdi
- Robotic Systems Laboratory, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Etienne Burdet
- Department of Bioengineering, Imperial College of Science, Technology and Medicine, London, United Kingdom
| | - Mohamed Bouri
- Robotic Systems Laboratory, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| | - Sharifa Himidan
- Pediatric General and Thoracic Surgery, Hospital for Sick Children, University of Toronto, Toronto, Ontario, Canada
| | - Hannes Bleuler
- Robotic Systems Laboratory, École Polytechnique Fédérale de Lausanne (EPFL), Lausanne, Switzerland
| |
Collapse
|
18
|
Robust Hand Motion Tracking through Data Fusion of 5DT Data Glove and Nimble VR Kinect Camera Measurements. SENSORS 2015; 15:31644-71. [PMID: 26694395 PMCID: PMC4721788 DOI: 10.3390/s151229868] [Citation(s) in RCA: 27] [Impact Index Per Article: 3.0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 08/27/2015] [Revised: 11/17/2015] [Accepted: 12/07/2015] [Indexed: 12/11/2022]
Abstract
Vision based interfaces for human computer interaction have gained increasing attention over the past decade. This study presents a data fusion approach of the Nimble VR vision based system, using the Kinect camera, with the contact based 5DT Data Glove. Data fusion was achieved through a Kalman filter. The Nimble VR and filter output were compared using measurements performed on (1) a wooden hand model placed in various static postures and orientations; and (2) three differently sized human hands during active finger flexions. Precision and accuracy of joint angle estimates as a function of hand posture and orientation were determined. Moreover, in light of possible self-occlusions of the fingers in the Kinect camera images, data completeness was assessed. Results showed that the integration of the Data Glove through the Kalman filter provided for the proximal interphalangeal (PIP) joints of the fingers a substantial improvement of 79% in precision, from 2.2 deg to 0.9 deg. Moreover, a moderate improvement of 31% in accuracy (being the mean angular deviation from the true joint angle) was established, from 24 deg to 17 deg. The metacarpophalangeal (MCP) joint was relatively unaffected by the Kalman filter. Moreover, the Data Glove increased data completeness, thus providing a substantial advantage over the sole use of the Nimble VR system.
Collapse
|