Control of a Supernumerary Robotic Hand by Foot: An Experimental Study in Virtual Reality

Human motor augmentation with an extra robotic arm without functional interference

Extra robotic arms (XRAs) are gaining interest in neuroscience and robotics, offering potential tools for daily activities. However, this compelling opportunity poses new challenges for sensorimotor control strategies and human-machine interfaces (HMIs). A key unsolved challenge is allowing users to proficiently control XRAs without hindering their existing functions. To address this, we propose a pipeline to identify suitable HMIs given a defined task to accomplish with the XRA. Following such a scheme, we assessed a multimodal motor HMI based on gaze detection and diaphragmatic respiration in a purposely designed modular neurorobotic platform integrating virtual reality and a bilateral upper limb exoskeleton. Our results show that the proposed HMI does not interfere with speaking or visual exploration and that it can be used to control an extra virtual arm independently from the biological ones or in coordination with them. Participants showed significant improvements in performance with daily training and retention of learning, with no further improvements when artificial haptic feedback was provided. As a final proof of concept, naïve and experienced participants used a simplified version of the HMI to control a wearable XRA. Our analysis indicates how the presented HMI can be effectively used to control XRAs. The observation that experienced users achieved a success rate 22.2% higher than that of naïve users, combined with the result that naïve users showed average success rates of 74% when they first engaged with the system, endorses the viability of both the virtual reality-based testing and training and the proposed pipeline.

Redundancy Resolution in Trimanual vs. Bimanual Tracking Tasks

Supernumerary limbs promise to allow users to perform complex tasks that would otherwise require the actions of teams. However, how the user's capability for multimanual coordination compares to bimanual coordination, and how the motor system decides to configure its limb contributions given task redundancy is unclear. We conducted bimanual and trimanual (with the foot as a third-hand controller) virtual reality visuomotor tracking experiments to study how 32 healthy participants changed their limb coordination in response to uninstructed cursor mapping changes. This used a shared cursor mapped to the average limbs' position for different limb combinations. The results show that most participants correctly identified the different mappings during bimanual tracking, and accordingly minimized task-irrelevant motion. Instead during trimanual coordination, participants consistently moved all three limbs concurrently, showing weaker ipsilateral hand-foot coordination. These findings show how redundancy resolution and the resulting coordination patterns differ between similar bimanual and trimanual tasks. Further research is needed to consider the effect of learning on coordination behaviour.

Comparing end-effector position and joint angle feedback for online robotic limb tracking

Somatosensation greatly increases the ability to control our natural body. This suggests that supplementing vision with haptic sensory feedback would also be helpful when a user aims at controlling a robotic arm proficiently. However, whether the position of the robot and its continuous update should be coded in a extrinsic or intrinsic reference frame is not known. Here we compared two different supplementary feedback contents concerning the status of a robotic limb in 2-DoFs configuration: one encoding the Cartesian coordinates of the end-effector of the robotic arm (i.e., Task-space feedback) and another and encoding the robot joints angles (i.e., Joint-space feedback). Feedback was delivered to blindfolded participants through vibrotactile stimulation applied on participants' leg. After a 1.5-hour training with both feedbacks, participants were significantly more accurate with Task compared to Joint-space feedback, as shown by lower position and aiming errors, albeit not faster (i.e., similar onset delay). However, learning index during training was significantly higher in Joint space feedback compared to Task-space feedback. These results suggest that Task-space feedback is probably more intuitive and more suited for activities which require short training sessions, while Joint space feedback showed potential for long-term improvement. We speculate that the latter, despite performing worse in the present work, might be ultimately more suited for applications requiring long training, such as the control of supernumerary robotic limbs for surgical robotics, heavy industrial manufacturing, or more generally, in the context of human movement augmentation.

Being an Avatar "for Real": A Survey on Virtual Embodiment in Augmented Reality

Virtual self-avatars have been increasingly used in Augmented Reality (AR) where one can see virtual content embedded into physical space. However, little is known about the perception of self-avatars in such a context. The possibility that their embodiment could be achieved in a similar way as in Virtual Reality opens the door to numerous applications in education, communication, entertainment, or the medical field. This article aims to review the literature covering the embodiment of virtual self-avatars in AR. Our goal is (i) to guide readers through the different options and challenges linked to the implementation of AR embodiment systems, (ii) to provide a better understanding of AR embodiment perception by classifying the existing knowledge, and (iii) to offer insight on future research topics and trends for AR and avatar research. To do so, we introduce a taxonomy of virtual embodiment experiences by defining a "body avatarization" continuum. The presented knowledge suggests that the sense of embodiment evolves in the same way in AR as in other settings, but this possibility has yet to be fully investigated. We suggest that, whilst it is yet to be well understood, the embodiment of avatars has a promising future in AR and conclude by discussing possible directions for research.

Principles of human movement augmentation and the challenges in making it a reality

Augmenting the body with artificial limbs controlled concurrently to one's natural limbs has long appeared in science fiction, but recent technological and neuroscientific advances have begun to make this possible. By allowing individuals to achieve otherwise impossible actions, movement augmentation could revolutionize medical and industrial applications and profoundly change the way humans interact with the environment. Here, we construct a movement augmentation taxonomy through what is augmented and how it is achieved. With this framework, we analyze augmentation that extends the number of degrees-of-freedom, discuss critical features of effective augmentation such as physiological control signals, sensory feedback and learning as well as application scenarios, and propose a vision for the field.

Enhanced location tracking in sensor fusion-assisted virtual reality micro-manipulation environments

Virtual reality (VR) technology plays a significant role in many biomedical applications. These VR scenarios increase the valuable experience of tasks requiring great accuracy with human subjects. Unfortunately, commercial VR controllers have large positioning errors in a micro-manipulation task. Here, we propose a VR-based framework along with a sensor fusion algorithm to improve the microposition tracking performance of a microsurgical tool. To the best of our knowledge, this is the first application of Kalman filter in a millimeter scale VR environment, by using the position data between the VR controller and an inertial measuring device. This study builds and tests two cases: (1) without sensor fusion tracking and (2) location tracking with active sensor fusion. The static and dynamic experiments demonstrate that the Kalman filter can provide greater precision during micro-manipulation in small scale VR scenarios.

Playing the piano with a robotic third thumb: assessing constraints of human augmentation

Contemporary robotics gives us mechatronic capabilities for augmenting human bodies with extra limbs. However, how our motor control capabilities pose limits on such augmentation is an open question. We developed a Supernumerary Robotic 3rd Thumbs (SR3T) with two degrees-of-freedom controlled by the user’s body to endow them with an extra contralateral thumb on the hand. We demonstrate that a pianist can learn to play the piano with 11 fingers within an hour. We then evaluate 6 naïve and 6 experienced piano players in their prior motor coordination and their capability in piano playing with the robotic augmentation. We show that individuals’ augmented performance with the SR3T could be explained by our new custom motor coordination assessment, the Human Augmentation Motor Coordination Assessment (HAMCA) performed pre-augmentation. Our work demonstrates how supernumerary robotics can augment humans in skilled tasks and that individual differences in their augmentation capability are explainable by their individual motor coordination abilities.

Human performance in three-hands tasks

The successful completion of complex tasks like hanging a picture or laparoscopic surgery requires coordinated motion of more than two limbs. User-controlled supernumerary robotic limbs (SL) have been proposed to bypass the need for coordination with a partner in such tasks. However, neither the capability to control multiple limbs alone relative to collaborative control with partners, nor how that capability varies across different tasks, is well understood. In this work, we present an investigation of tasks requiring three-hands where the foot was used as an additional source of motor commands. We considered: (1) how does simultaneous control of three hands compare to a cooperating dyad; (2) how this relative performance was altered by the existence of constraints emanating from real or virtual physical connections (mechanical constraints) or from cognitive limits (cognitive constraints). It was found that a cooperating dyad outperformed a single user in all scenarios in terms of task score, path efficiency and motion smoothness. However, while the participants were able to reach more targets with increasing mechanical constraints/decreasing number of simultaneous goals, the relative difference in performance between a dyad and a participant performing trimanual activities decreased, suggesting further potential for SLs in this class of scenario.

Serious Game for Teaching Undergraduate Medical Students in Cleft lip and Palate Treatment Protocol

OBJECTIVES

To develop and evaluate a serious game to deliver knowledge about the cleft lip with or without cleft palate (CL/P) protocol used in Craniofacial Center, Chiang Mai University.

METHODS

The game "Cleft Island" was developed then extensively evaluated using different criteria, including the measured knowledge gained by participants, and gameplay experience. A group of 53 fourth and fifth-year medical students (male: 24, female: 29) were recruited to take part in an experiment. They were from the Faculty of Medicine, Chiang Mai University, and had just rotated to the Department of Surgery. Three evaluations were conducted: 1) to test whether the players had competently gained CL/P knowledge; 2) to evaluate the usability of the game according to the System Usability Scale (SUS); and 3) to evaluate the game experience in terms of a Game Experience Questionnaire (GEQ) test.

RESULTS

The results indicate a statistically significant improvement of medical students' knowledge after performing the Wilcoxon Signed-Ranks test (p < 0.5) between pre-test and post-test scores of the same medical student group; the acceptable average SUS score (M = 55.28) of the serious game; a moderate degree of experience of the GEQ components including positive affect (M = 2.64), competence (M = 2.49), and immersion (M = 2.21).

CONCLUSIONS

Cleft Island can be used as an effective supplementary instructional material, which has the potential to provide significant knowledge of CL/P treatment protocol for the players. As far as the authors are aware, this is the first study to implement and assess a serious game for training in CL/P protocol.

Concurrent control of a brain-computer interface and natural overt movements

OBJECTIVE

A primary control signal in brain-computer interfaces (BCIs) have been cortical signals related to movement. However, in cases where natural motor function remains, BCI control signals may interfere with other possibly simultaneous activity for useful ongoing movement. We sought to determine if the brain could learn to control both a BCI and concurrent overt movement execution in such cases.

APPROACH

We designed experiments where BCI and overt movements must be used concurrently and in coordination to achieve a 2D centre out control. Power in the 70-90 Hz band of human electrocorticography (ECoG) signals, was used to generate BCI control commands for vertical movement of the cursor. These signals were deliberately recorded from the same human cortical site that produced the strongest movement related activity associated with the concurrent overt finger movements required for the horizontal movement of the cursor.

MAIN RESULTS

We demonstrate that three subjects were able to perform the concurrent BCI task, controlling BCI and natural movements simultaneously and to a large extent independently. We conclude that the brain is capable of dissociating the original control signal dependency on movement, producing specific BCI control signals in the presence of motor related responses from the ongoing overt behaviour with which the BCI signal was initially correlated.

SIGNIFICANCE

We demonstrate a novel human brain-computer interface (BCI) which can be used to control movement concurrently and in coordination with movements of the natural limbs. This demonstrates the dissociation of cortical activity from the behaviour with which it was originally associated despite the ongoing behaviour and shows the feasibility of achieving simultaneous BCI control of devices with natural movements.

A magnetic compatible supernumerary robotic finger for functional magnetic resonance imaging (fMRI) acquisitions: Device description and preliminary results

The Supernumerary robotic limbs are a recently introduced class of wearable robots that, differently from traditional prostheses and exoskeletons, aim at adding extra effectors (i.e., arms, legs, or fingers) to the human user, rather than substituting or enhancing the natural ones. However, it is still undefined whether the use of supernumerary robotic limbs could specifically lead to neural modifications in brain dynamics. The illusion of owning the part of body has been already proven in many experimental observations, such as those relying on multisensory integration (e.g., rubber hand illusion), prosthesis and even on virtual reality. In this paper we present a description of a novel magnetic compatible supernumerary robotic finger together with preliminary observations from two functional magnetic resonance imaging (fMRI) experiments, in which brain activity was measured before and after a period of training with the robotic device, and during the use of the novel MRI-compatible version of the supernumerary robotic finger. Results showed that the usage of the MR-compatible robotic finger is safe and does not produce artifacts on MRI images. Moreover, the training with the supernumerary robotic finger recruits a network of motor-related cortical regions (i.e. primary and supplementary motor areas), hence the same motor network of a fully physiological voluntary motor gestures.

Positioning the endoscope in laparoscopic surgery by foot: Influential factors on surgeons' performance in virtual trainer

We have investigated how surgeons can use the foot to position a laparoscopic endoscope, a task that normally requires an extra assistant. Surgeons need to train in order to exploit the possibilities offered by this new technique and safely manipulate the endoscope together with the hands movements. A realistic abdominal cavity has been developed as training simulator to investigate this multi-arm manipulation. In this virtual environment, the surgeon's biological hands are modelled as laparoscopic graspers while the viewpoint is controlled by the dominant foot. 23 surgeons and medical students performed single-handed and bimanual manipulation in this environment. The results show that residents had superior performance compared to both medical students and more experienced surgeons, suggesting that residency is an ideal period for this training. Performing the single-handed task improves the performance in the bimanual task, whereas the converse was not true.

Ownership and Agency of an Independent Supernumerary Hand Induced by an Imitation Brain-Computer Interface

To study body ownership and control, illusions that elicit these feelings in non-body objects are widely used. Classically introduced with the Rubber Hand Illusion, these illusions have been replicated more recently in virtual reality and by using brain-computer interfaces. Traditionally these illusions investigate the replacement of a body part by an artificial counterpart, however as brain-computer interface research develops it offers us the possibility to explore the case where non-body objects are controlled in addition to movements of our own limbs. Therefore we propose a new illusion designed to test the feeling of ownership and control of an independent supernumerary hand. Subjects are under the impression they control a virtual reality hand via a brain-computer interface, but in reality there is no causal connection between brain activity and virtual hand movement but correct movements are observed with 80% probability. These imitation brain-computer interface trials are interspersed with movements in both the subjects' real hands, which are in view throughout the experiment. We show that subjects develop strong feelings of ownership and control over the third hand, despite only receiving visual feedback with no causal link to the actual brain signals. Our illusion is crucially different from previously reported studies as we demonstrate independent ownership and control of the third hand without loss of ownership in the real hands.

In a demanding task, three-handed manipulation is preferred to two-handed manipulation

Equipped with a third hand under their direct control, surgeons may be able to perform certain surgical interventions alone; this would reduce the need for a human assistant and related coordination difficulties. However, does human performance improve with three hands compared to two hands? To evaluate this possibility, we carried out a behavioural study on the performance of naive adults catching objects with three virtual hands controlled by their two hands and right foot. The subjects could successfully control the virtual hands in a few trials. With this control strategy, the workspace of the hands was inversely correlated with the task velocity. The comparison of performance between the three and two hands control revealed no significant difference of success in catching falling objects and in average effort during the tasks. Subjects preferred the three handed control strategy, found it easier, with less physical and mental burden. Although the coordination of the foot with the natural hands increased trial after trial, about two minutes of practice was not sufficient to develop a sense of ownership towards the third arm.

Robust Hand Motion Tracking through Data Fusion of 5DT Data Glove and Nimble VR Kinect Camera Measurements

Vision based interfaces for human computer interaction have gained increasing attention over the past decade. This study presents a data fusion approach of the Nimble VR vision based system, using the Kinect camera, with the contact based 5DT Data Glove. Data fusion was achieved through a Kalman filter. The Nimble VR and filter output were compared using measurements performed on (1) a wooden hand model placed in various static postures and orientations; and (2) three differently sized human hands during active finger flexions. Precision and accuracy of joint angle estimates as a function of hand posture and orientation were determined. Moreover, in light of possible self-occlusions of the fingers in the Kinect camera images, data completeness was assessed. Results showed that the integration of the Data Glove through the Kalman filter provided for the proximal interphalangeal (PIP) joints of the fingers a substantial improvement of 79% in precision, from 2.2 deg to 0.9 deg. Moreover, a moderate improvement of 31% in accuracy (being the mean angular deviation from the true joint angle) was established, from 24 deg to 17 deg. The metacarpophalangeal (MCP) joint was relatively unaffected by the Kalman filter. Moreover, the Data Glove increased data completeness, thus providing a substantial advantage over the sole use of the Nimble VR system.