Exploring a GPT-based large language model for variable autonomy in a VR-based human-robot teaming simulation

In a rapidly evolving digital landscape autonomous tools and robots are becoming commonplace. Recognizing the significance of this development, this paper explores the integration of Large Language Models (LLMs) like Generative pre-trained transformer (GPT) into human-robot teaming environments to facilitate variable autonomy through the means of verbal human-robot communication. In this paper, we introduce a novel simulation framework for such a GPT-powered multi-robot testbed environment, based on a Unity Virtual Reality (VR) setting. This system allows users to interact with simulated robot agents through natural language, each powered by individual GPT cores. By means of OpenAI's function calling, we bridge the gap between unstructured natural language input and structured robot actions. A user study with 12 participants explores the effectiveness of GPT-4 and, more importantly, user strategies when being given the opportunity to converse in natural language within a simulated multi-robot environment. Our findings suggest that users may have preconceived expectations on how to converse with robots and seldom try to explore the actual language and cognitive capabilities of their simulated robot collaborators. Still, those users who did explore were able to benefit from a much more natural flow of communication and human-like back-and-forth. We provide a set of lessons learned for future research and technical implementations of similar systems.

Exploring Control Authority Preferences in Robotic Arm Assistance for Power Wheelchair Users

This paper uses mixed methods to explore the preliminary design of control authority preferences for an Assistive Robotic Manipulator (ARM). To familiarize users with an intelligent robotic arm, we perform two kitchen task iterations: one with user-initiated software autonomy (predefined autonomous actions) and one with manual control. Then, we introduce a third scenario, enabling users to choose between manual control and system delegation throughout the task. Results showed that, while manually switching modes and controlling the arm via joystick had a higher mental workload, participants still preferred full joystick control. Thematic analysis indicates manual control offered greater freedom and sense of accomplishment. Participants reacted positively to the idea of an interactive assistive system. Users did not want to ask the system to only assist, by taking over for certain actions, but also asked for situational feedback (e.g., 'How close am I (the gripper)?', 'Is the lid centered over the jug?'). This speaks to a future assistive system that ensures the user feels like they drive the system for the entirety of the task and provides action collaboration in addition to more granular situational awareness feedback.

Overview of the role of robots in upper limb disabilities rehabilitation: a scoping review

BACKGROUND

Neuromotor rehabilitation and improvement of upper limb functions are necessary to improve the life quality of patients who have experienced injuries or have pathological outcomes. Modern approaches, such as robotic-assisted rehabilitation can help to improve rehabilitation processes and thus improve upper limb functions. Therefore, the aim of this study was to investigate the role of robots in upper limb disability improvement and rehabilitation.

METHODS

This scoping review was conducted by search in PubMed, Web of Science, Scopus, and IEEE (January 2012- February 2022). Articles related to upper limb rehabilitation robots were selected. The methodological quality of all the included studies will be appraised using the Mixed Methods Appraisal Tool (MMAT). We used an 18-field data extraction form to extract data from articles and extracted the information such as study year, country, type of study, purpose, illness or accident leading to disability, level of disability, assistive technologies, number of participants in the study, sex, age, rehabilitated part of the upper limb using a robot, duration and frequency of treatment, methods of performing rehabilitation exercises, type of evaluation, number of participants in the evaluation process, duration of intervention, study outcomes, and study conclusions. The selection of articles and data extraction was made by three authors based on inclusion and exclusion criteria. Disagreements were resolved through consultation with the fifth author. Inclusion criteria were articles involving upper limb rehabilitation robots, articles about upper limb disability caused by any illness or injury, and articles published in English. Also, articles involving other than upper limb rehabilitation robots, robots related to rehabilitation of diseases other than upper limb, systematic reviews, reviews, and meta-analyses, books, book chapters, letters to the editor, and conference papers were also excluded. Descriptive statistics methods (frequency and percentage) were used to analyses the data.

RESULTS

We finally included 55 relevant articles. Most of the studies were done in Italy (33.82%). Most robots were used to rehabilitate stroke patients (80%). About 60.52% of the studies used games and virtual reality rehabilitate the upper limb disabilities using robots. Among the 14 types of applied evaluation methods, "evaluation and measurement of upper limb function and dexterity" was the most applied evaluation method. "Improvement in musculoskeletal functions", "no adverse effect on patients", and "Safe and reliable treatment" were the most cited outcomes, respectively.

CONCLUSIONS

Our findings show that robots can improve musculoskeletal functions (musculoskeletal strength, sensation, perception, vibration, muscle coordination, less spasticity, flexibility, and range of motion) and empower people by providing a variety of rehabilitation capabilities.

Discernment on assistive technology for the care and support requirements of older adults and differently-abled individuals

Assistive technology for the differently abled and older adults has made remarkable achievements in providing rehabilitative, adaptive, and assistive devices. It provides huge assistance for people with physical impairments to lead a better self-reliant daily life, in terms of mobility, education, rehabilitation, etc. This technology ranges from simple hand-held devices to complex robotic accessories which promote the individual's independence. This study aimed at identifying the assistance required by differently-abled individuals, and the solutions proposed by different researchers, and reviewed their merits and demerits. It provides a detailed discussion on the state of art assistive technologies, their applications, challenges, types, and their usage for rehabilitation. The study also identifies different unexplored research areas related to assistive technology that can improve the daily life of individuals and advance the field. Despite their high usage, assistive technologies have some limitations which have been briefly described in the study. This review, therefore, can help understand the utilization, and pros and cons of assistive devices in rehabilitation engineering and assistive technologies.

User Based Development and Test of the EXOTIC Exoskeleton: Empowering Individuals with Tetraplegia Using a Compact, Versatile, 5-DoF Upper Limb Exoskeleton Controlled through Intelligent Semi-Automated Shared Tongue Control

This paper presents the EXOTIC- a novel assistive upper limb exoskeleton for individuals with complete functional tetraplegia that provides an unprecedented level of versatility and control. The current literature on exoskeletons mainly focuses on the basic technical aspects of exoskeleton design and control while the context in which these exoskeletons should function is less or not prioritized even though it poses important technical requirements. We considered all sources of design requirements, from the basic technical functions to the real-world practical application. The EXOTIC features: (1) a compact, safe, wheelchair-mountable, easy to don and doff exoskeleton capable of facilitating multiple highly desired activities of daily living for individuals with tetraplegia; (2) a semi-automated computer vision guidance system that can be enabled by the user when relevant; (3) a tongue control interface allowing for full, volitional, and continuous control over all possible motions of the exoskeleton. The EXOTIC was tested on ten able-bodied individuals and three users with tetraplegia caused by spinal cord injury. During the tests the EXOTIC succeeded in fully assisting tasks such as drinking and picking up snacks, even for users with complete functional tetraplegia and the need for a ventilator. The users confirmed the usability of the EXOTIC.

Assistive Robots for Patients With Amyotrophic Lateral Sclerosis: Exploratory Task-Based Evaluation Study With an Early-Stage Demonstrator

Background

Although robotic manipulators have great potential in promoting motor independence of people with motor impairments, only few systems are currently commercially available. In addition to technical, economic, and normative barriers, a key challenge for their distribution is the current lack of evidence regarding their usefulness, acceptance, and user-specific requirements.

Objective

Against this background, a semiautonomous robot system was developed in the research and development project, robot-assisted services for individual and resource-oriented intensive and palliative care of people with amyotrophic lateral sclerosis (ROBINA), to support people with amyotrophic lateral sclerosis (ALS) in various everyday activities.

Methods

The developed early-stage demonstrator was evaluated in a task-based laboratory study of 11 patients with ALS. On the basis of a multimethod design consisting of standardized questionnaires, open-ended questions, and observation protocols, participants were asked about its relevance to everyday life, usability, and design requirements.

Results

Most participants considered the system to provide relevant support within the test scenarios and for their everyday life. On the basis of the System Usability Scale, the overall usability of the robot-assisted services for individual and resource-oriented intensive and palliative care of people with ALS system was rated as excellent, with a median of 90 (IQR 75-95) points. Moreover, 3 central areas of requirements for the development of semiautonomous robotic manipulators were identified and discussed: requirements for semiautonomous human-robot collaboration, requirements for user interfaces, and requirements for the adaptation of robotic capabilities regarding everyday life.

Conclusions

Robotic manipulators can contribute to increase the autonomy of people with ALS. A key issue for future studies is how the existing ability level and the required robotic capabilities can be balanced to ensure both high user satisfaction and effective and efficient task performance.

Assistive Technology in Multiple Sclerosis Patients—Two Points of View

Objective: The goal of our study was determining the current needs and acceptance of patients with multiple sclerosis (MS) in the field of assistive technologies using materials from the “RAMCIP” project (Robotic Assistant for Mild Cognitive Impairment Patient at Home). Methods: There were two target groups: a population with MS, and medical personnel experienced in treating MS patients. This study was based on a two-step design method (workshops and surveys). Using the Likert scale, we identified the prioritization of users’ needs. Additionally, demographic and disease-specific data and their correlations with each other and with the level of priority of functionality were analyzed. Moreover, the acceptance aspect of the assistant robot and the respondents’ readiness to use it were determined. Results: We gathered 307 completed surveys (176 from MS patients, 131 from medical personnel). Functional capabilities from the safety category were a high priority in most cases. The medium priority functions concerned daily activities that required physical assistance and home management. The differences in prioritization between the two groups were also found. Variables such as age, level of disability, cognitive impairment, depression, and fatigue were associated with the priority level of the functionalities. Conclusion: In summary, our findings might contribute to a better adaptation of robotic assistants to the needs and expectations of the MS population.

Manual 3D Control of an Assistive Robotic Manipulator Using Alpha Rhythms and an Auditory Menu: A Proof-of-Concept

Brain–Computer Interfaces (BCIs) have been regarded as potential tools for individuals with severe motor disabilities, such as those with amyotrophic lateral sclerosis, that render interfaces that rely on movement unusable. This study aims to develop a dependent BCI system for manual end-point control of a robotic arm. A proof-of-concept system was devised using parieto-occipital alpha wave modulation and a cyclic menu with auditory cues. Users choose a movement to be executed and asynchronously stop said action when necessary. Tolerance intervals allowed users to cancel or confirm actions. Eight able-bodied subjects used the system to perform a pick-and-place task. To investigate the potential learning effects, the experiment was conducted twice over the course of two consecutive days. Subjects obtained satisfactory completion rates (84.0 ± 15.0% and 74.4 ± 34.5% for the first and second day, respectively) and high path efficiency (88.9 ± 11.7% and 92.2 ± 9.6%). Subjects took on average 439.7 ± 203.3 s to complete each task, but the robot was only in motion 10% of the time. There was no significant difference in performance between both days. The developed control scheme provided users with intuitive control, but a considerable amount of time is spent waiting for the right target (auditory cue). Implementing other brain signals may increase its speed.

Computer Vision-Based Adaptive Semi-Autonomous Control of an Upper Limb Exoskeleton for Individuals with Tetraplegia

We propose the use of computer vision for adaptive semi-autonomous control of an upper limb exoskeleton for assisting users with severe tetraplegia to increase independence and quality of life. A tongue-based interface was used together with the semi-autonomous control such that individuals with complete tetraplegia were able to use it despite being paralyzed from the neck down. The semi-autonomous control uses computer vision to detect nearby objects and estimate how to grasp them to assist the user in controlling the exoskeleton. Three control schemes were tested: non-autonomous (i.e., manual control using the tongue) control, semi-autonomous control with a fixed level of autonomy, and a semi-autonomous control with a confidence-based adaptive level of autonomy. Studies on experimental participants with and without tetraplegia were carried out. The control schemes were evaluated both in terms of their performance, such as the time and number of commands needed to complete a given task, as well as ratings from the users. The studies showed a clear and significant improvement in both performance and user ratings when using either of the semi-autonomous control schemes. The adaptive semi-autonomous control outperformed the fixed version in some scenarios, namely, in the more complex tasks and with users with more training in using the system.

The role of user preference in the customized control of robotic exoskeletons

User preference is a promising objective for the control of robotic exoskeletons because it may capture the multifactorial nature of exoskeleton use. However, to use it, we must first understand its characteristics in the context of exoskeleton control. Here, we systematically measured the control preferences of individuals wearing bilateral ankle exoskeletons during walking. We investigated users' repeatability identifying their preferences and how preference changes with walking speed, device exposure, and between individuals with different technical backgrounds. Twelve naive and 12 knowledgeable nondisabled participants identified their preferred assistance in repeated trials by simultaneously self-tuning the magnitude and timing of peak torque. They were blinded to the control parameters and relied solely on their perception of the assistance to guide their tuning. We found that participants' preferences ranged from 7.9 to 19.4 newton-meters and 54.1 to 59.2 percent of the gait cycle. Across trials, participants repeatably identified their preferences with a mean standard deviation of 1.7 newton-meters and 1.5 percent of the gait cycle. Within a trial, participants converged on their preference in 105 seconds. As the experiment progressed, naive users preferred higher torque magnitude. At faster walking speeds, these individuals were more precise at identifying the magnitude of their preferred assistance. Knowledgeable users preferred higher torque than naive users. These results highlight that although preference is a dynamic quantity, individuals can reliably identify their preferences. This work motivates strategies for the control of lower limb exoskeletons in which individuals customize assistance according to their unique preferences and provides meaningful insight into how users interact with exoskeletons.

Evaluating the task effectiveness and user satisfaction with different operation modes of an assistive bathing robot in older adults

Bathing robots have the potential to foster the independence of older adults who require assistance with bathing. Making human-robot interaction (HRI) for older persons as easy, effective, and user-satisfying as possible is, however, a major challenge in the development of such robots. The study aimed to evaluate the effectiveness (coverage, step effectiveness) and user satisfaction (After-Scenario Questionnaire, ASQ) with three operation modes (autonomous operation, shared control, tele-manipulation) for the HRI with a bathing robot in potential users. Twenty-five older adults who require bathing assistance tested these operation modes in a water rinsing task for the upper back. Autonomous operation led to maximum effectiveness (100%), which was significantly worse in the shared control (51.6-79.4%, p ≤ 0.001) and tele-manipulation mode (43.9-64.4%, p < .001). In the user-controlled modes, effectiveness decreased with decreasing robot assistance (shared control: 51.6-79.4% vs. tele-manipulation: 43.9-64.4%, p = 0.009-0.016). User satisfaction with the autonomous operation (ASQ: 2.0 ± 1.0pt.) was higher than with the tele-manipulation mode (ASQ: 3.0 ± 1.4pt., p = 0.003) and in trend also than with the shared control mode (ASQ: 2.5 ± 1.5pt., p = 0.071). Our study suggests that for an effective and highly satisfying HRI with a bathing robot in older users, operation modes with high robot autonomy requiring a minimum of user input seem to be necessary.

Adapt or Perish? Exploring the Effectiveness of Adaptive DoF Control Interaction Methods for Assistive Robot Arms

Robot arms are one of many assistive technologies used by people with motor impairments. Assistive robot arms can allow people to perform activities of daily living (ADL) involving grasping and manipulating objects in their environment without the assistance of caregivers. Suitable input devices (e.g., joysticks) mostly have two Degrees of Freedom (DoF), while most assistive robot arms have six or more. This results in time-consuming and cognitively demanding mode switches to change the mapping of DoFs to control the robot. One option to decrease the difficulty of controlling a high-DoF assistive robot arm using a low-DoF input device is to assign different combinations of movement-DoFs to the device’s input DoFs depending on the current situation (adaptive control). To explore this method of control, we designed two adaptive control methods for a realistic virtual 3D environment. We evaluated our methods against a commonly used non-adaptive control method that requires the user to switch controls manually. This was conducted in a simulated remote study that used Virtual Reality and involved 39 non-disabled participants. Our results show that the number of mode switches necessary to complete a simple pick-and-place task decreases significantly when using an adaptive control type. In contrast, the task completion time and workload stay the same. A thematic analysis of qualitative feedback of our participants suggests that a longer period of training could further improve the performance of adaptive control methods.

My Caregiver the Cobot: Comparing Visualization Techniques to Effectively Communicate Cobot Perception to People with Physical Impairments

Nowadays, robots are found in a growing number of areas where they collaborate closely with humans. Enabled by lightweight materials and safety sensors, these cobots are gaining increasing popularity in domestic care, where they support people with physical impairments in their everyday lives. However, when cobots perform actions autonomously, it remains challenging for human collaborators to understand and predict their behavior, which is crucial for achieving trust and user acceptance. One significant aspect of predicting cobot behavior is understanding their perception and comprehending how they “see” the world. To tackle this challenge, we compared three different visualization techniques for Spatial Augmented Reality. All of these communicate cobot perception by visually indicating which objects in the cobot’s surrounding have been identified by their sensors. We compared the well-established visualizations Wedge and Halo against our proposed visualization Line in a remote user experiment with participants suffering from physical impairments. In a second remote experiment, we validated these findings with a broader non-specific user base. Our findings show that Line, a lower complexity visualization, results in significantly faster reaction times compared to Halo, and lower task load compared to both Wedge and Halo. Overall, users prefer Line as a more straightforward visualization. In Spatial Augmented Reality, with its known disadvantage of limited projection area size, established off-screen visualizations are not effective in communicating cobot perception and Line presents an easy-to-understand alternative.

Eyes-Free Tongue Gesture and Tongue Joystick Control of a Five DOF Upper-Limb Exoskeleton for Severely Disabled Individuals

Spinal cord injury can leave the affected individual severely disabled with a low level of independence and quality of life. Assistive upper-limb exoskeletons are one of the solutions that can enable an individual with tetraplegia (paralysis in both arms and legs) to perform simple activities of daily living by mobilizing the arm. Providing an efficient user interface that can provide full continuous control of such a device—safely and intuitively—with multiple degrees of freedom (DOFs) still remains a challenge. In this study, a control interface for an assistive upper-limb exoskeleton with five DOFs based on an intraoral tongue-computer interface (ITCI) for individuals with tetraplegia was proposed. Furthermore, we evaluated eyes-free use of the ITCI for the first time and compared two tongue-operated control methods, one based on tongue gestures and the other based on dynamic virtual buttons and a joystick-like control. Ten able-bodied participants tongue controlled the exoskeleton for a drinking task with and without visual feedback on a screen in three experimental sessions. As a baseline, the participants performed the drinking task with a standard gamepad. The results showed that it was possible to control the exoskeleton with the tongue even without visual feedback and to perform the drinking task at 65.1% of the speed of the gamepad. In a clinical case study, an individual with tetraplegia further succeeded to fully control the exoskeleton and perform the drinking task only 5.6% slower than the able-bodied group. This study demonstrated the first single-modal control interface that can enable individuals with complete tetraplegia to fully and continuously control a five-DOF upper limb exoskeleton and perform a drinking task after only 2 h of training. The interface was used both with and without visual feedback.

Continuous Tongue Robot Mapping for Paralyzed Individuals Improves the Functional Performance of Tongue-Based Robotic Assistance

Individuals with tetraplegia have a challenging life due to a lack of independence and autonomy. Assistive robots have the potential to assist with the activities of daily living and thus improve the quality of life. However, an efficient and reliable control interface for severely disabled individuals is still missing. An intraoral tongue-computer interface (ITCI) for people with tetraplegia has previously been introduced and tested for controlling a robotic manipulator in a study deploying discrete tongue robot mapping. To improve the efficiency of the interface, the current study proposed the use of virtual buttons based on the ITCI and evaluated them in combination with a joystick-like control implementation, enabling continuous control commands. Twelve able-bodied volunteers participated in a three-day experiment. They controlled an assistive robotic manipulator through the tongue to perform two tasks: Pouring water in a cup (PW) and picking up a roll of tape (PUT). Four different tongue-robot mapping methods were compared. The results showed that using continuous commands reduced the task completion time by 16% and the number of commands of the PUT test by 20% compared with discrete commands. The highest success rate for completing the tasks was 77.8% for the PUT test and 100% for the PW test, both achieved by the control methods with continuous commands. Thus, the study demonstrated that incorporating continuous commands can improve the performance of the ITCI system for controlling robotic manipulators.

Toward Shared Autonomy Control Schemes for Human-Robot Systems: Action Primitive Recognition Using Eye Gaze Features

The functional independence of individuals with upper limb impairment could be enhanced by teleoperated robots that can assist with activities of daily living. However, robot control is not always intuitive for the operator. In this work, eye gaze was leveraged as a natural way to infer human intent and advance action recognition for shared autonomy control schemes. We introduced a classifier structure for recognizing low-level action primitives that incorporates novel three-dimensional gaze-related features. We defined an action primitive as a triplet comprised of a verb, target object, and hand object. A recurrent neural network was trained to recognize a verb and target object, and was tested on three different activities. For a representative activity (making a powdered drink), the average recognition accuracy was 77% for the verb and 83% for the target object. Using a non-specific approach to classifying and indexing objects in the workspace, we observed a modest level of generalizability of the action primitive classifier across activities, including those for which the classifier was not trained. The novel input features of gaze object angle and its rate of change were especially useful for accurately recognizing action primitives and reducing the observational latency of the classifier.

AMiCUS 2.0-System Presentation and Demonstration of Adaptability to Personal Needs by the Example of an Individual with Progressed Multiple Sclerosis

AMiCUS is a human–robot interface that enables tetraplegics to control an assistive robotic arm in real-time using only head motion, allowing them to perform simple manipulation tasks independently. The interface may be used as a standalone system or to provide direct control as part of a semi-autonomous system. Within this work, we present our new gesture-free prototype AMiCUS 2.0, which has been designed with special attention to accessibility and ergonomics. As such, AMiCUS 2.0 addresses the needs of tetraplegics with additional impairments that may come along with multiple sclerosis. In an experimental setup, both AMiCUS 1.0 and 2.0 are compared with each other, showing higher accessibility and usability for AMiCUS 2.0. Moreover, in an activity of daily living, a proof-of-concept is provided that an individual with progressed multiple sclerosis is able to operate the robotic arm in a temporal and functional scope, as would be necessary to perform direct control tasks for use in a commercial semi-autonomous system. The results indicate that AMiCUS 2.0 makes an important step towards closing the gaps of assistive technology, being accessible to those who rely on such technology the most.

Probabilistic Human Intent Recognition for Shared Autonomy in Assistive Robotics

Effective human-robot collaboration in shared autonomy requires reasoning about the intentions of the human partner. To provide meaningful assistance, the autonomy has to first correctly predict, or infer, the intended goal of the human collaborator. In this work, we present a mathematical formulation for intent inference during assistive teleoperation under shared autonomy. Our recursive Bayesian filtering approach models and fuses multiple non-verbal observations to probabilistically reason about the intended goal of the user without explicit communication. In addition to contextual observations, we model and incorporate the human agent's behavior as goal-directed actions with adjustable rationality to inform intent recognition. Furthermore, we introduce a user-customized optimization of this adjustable rationality to achieve user personalization. We validate our approach with a human subjects study that evaluates intent inference performance under a variety of goal scenarios and tasks. Importantly, the studies are performed using multiple control interfaces that are typically available to users in the assistive domain, which differ in the continuity and dimensionality of the issued control signals. The implications of the control interface limitations on intent inference are analyzed. The study results show that our approach in many scenarios outperforms existing solutions for intent inference in assistive teleoperation, and otherwise performs comparably. Our findings demonstrate the benefit of probabilistic modeling and the incorporation of human agent behavior as goal-directed actions where the adjustable rationality model is user customized. Results further show that the underlying intent inference approach directly affects shared autonomy performance, as do control interface limitations.

Semi-Autonomous Robotic Arm Reaching With Hybrid Gaze-Brain Machine Interface

Recent developments in the non-muscular human-robot interface (HRI) and shared control strategies have shown potential for controlling the assistive robotic arm by people with no residual movement or muscular activity in upper limbs. However, most non-muscular HRIs only produce discrete-valued commands, resulting in non-intuitive and less effective control of the dexterous assistive robotic arm. Furthermore, the user commands and the robot autonomy commands usually switch in the shared control strategies of such applications. This characteristic has been found to yield a reduced sense of agency as well as frustration for the user according to previous user studies. In this study, we firstly propose an intuitive and easy-to-learn-and-use hybrid HRI by combing the Brain-machine interface (BMI) and the gaze-tracking interface. For the proposed hybrid gaze-BMI, the continuous modulation of the movement speed via the motor intention occurs seamlessly and simultaneously to the unconstrained movement direction control with the gaze signals. We then propose a shared control paradigm that always combines user input and the autonomy with the dynamic combination regulation. The proposed hybrid gaze-BMI and shared control paradigm were validated for a robotic arm reaching task performed with healthy subjects. All the users were able to employ the hybrid gaze-BMI for moving the end-effector sequentially to reach the target across the horizontal plane while also avoiding collisions with obstacles. The shared control paradigm maintained as much volitional control as possible, while providing the assistance for the most difficult parts of the task. The presented semi-autonomous robotic system yielded continuous, smooth, and collision-free motion trajectories for the end effector approaching the target. Compared to a system without assistances from robot autonomy, it significantly reduces the rate of failure as well as the time and effort spent by the user to complete the tasks.

A Bayesian Shared Control Approach for Wheelchair Robot With Brain Machine Interface

To enhance the performance of the brain-actuated robot system, a novel shared controller based on Bayesian approach is proposed for intelligently combining robot automatic control and brain-actuated control, which takes into account the uncertainty of robot perception, action and human control. Based on maximum a posteriori probability (MAP), this method establishes the probabilistic models of human and robot control commands to realize the optimal control of a brain-actuated shared control system. Application on an intelligent Bayesian shared control system based on steady-state visual evoked potential (SSVEP)-based brain machine interface (BMI) is presented for all-time continuous wheelchair navigation task. Moreover, to obtain more accurate brain control commands for shared controller and adapt the proposed system to the uncertainty of electroencephalogram (EEG), a hierarchical brain control mechanism with feedback rule is designed. Experiments have been conducted to verify the proposed system in several scenarios. Eleven subjects participated in our experiments and the results illustrate the effectiveness of the proposed method.

Impacts of robotic arm use on individuals with upper extremity disabilities: A scoping review

BACKGROUND.

Robotic arms may help users perform various activities. Even though robotic arms are commercially available, their impacts are still poorly understood.

PURPOSE.

This scoping review aimed to identify the potential impacts of using robotic arms for individuals with upper-extremity disabilities and appraise the scientific quality of the selected studies.

METHOD.

A search for studies published between 1970 and 2016 was conducted in PubMed, Embase, Compendex, and Scopus. The Canadian Model of Occupational Performance and Engagement was used to classify activities in which impacts were evaluated. The quality of each study was rated using McMaster University's critical review form for quantitative studies.

FINDINGS.

Thirty-six studies were reviewed, which evaluated self-care (21), productivity (33), and leisure (8). The short-term impacts were more commonly documented than long-term impacts. The impacts identified were mostly positive. The studies' mean quality score was 8.8/15.

IMPLICATIONS.

Additional studies with more rigorous conditions are needed to produce higher-quality scientific evidence of the long-term impacts of robotic arm use.

Robot Learning of Assistive Manipulation Tasks by Demonstration via Head Gesture-based Interface

Assistive robotic manipulators have the potential to support the lives of people suffering from severe motor impairments. They can support individuals with disabilities to independently perform daily living activities, such as drinking, eating, manipulation tasks, and opening doors. An attractive solution is to enable motor impaired users to teach a robot by providing demonstrations of daily living tasks. The user controls the robot 'manually' with an intuitive human-robot interface to provide demonstration, which is followed by the robot learning of the performed task. However, the control of robotic manipulators by motor impaired individuals is a challenging topic. In this paper, a novel head gesture-based interface for hands-free robot control and a framework for robot learning from demonstration are presented. The head gesture-based interface consists of a camera mounted on the user's hat, which records the changes in the viewed scene due to the head motion. The head gesture recognition is performed using the optical flow for feature extraction and support vector machine for gesture classification. The recognized head gestures are further mapped into robot control commands to perform object manipulation task. The robot learns the demonstrated task by generating the sequence of actions and Gaussian Mixture Model method is used to segment the demonstrated path of the robot's end-effector. During the robotic reproduction of the task, the modified Gaussian Mixture Model and Gaussian Mixture Regression are used to adapt to environmental changes. The proposed framework was evaluated in a real-world assistive robotic scenario in a small study involving 13 participants; 12 able-bodied and one tetraplegic. The presented results demonstrate a potential of the proposed framework to enable severe motor impaired individuals to demonstrate daily living tasks to robotic manipulators.

Semi-Autonomous Tongue Control of an Assistive Robotic Arm for Individuals with Quadriplegia

Individuals suffering from quadriplegia can achieve increased independence by using an assistive robotic manipulator (ARM). However, due to their disability, the interfaces that can be used to operate such devices become limited. A versatile intraoral tongue control interface (ITCI) has previously been develop for this user group, as the tongue is usually spared from disability. A previous study has shown that the ITCI can provide direct and continuous control of 6-7 degrees of freedom (DoF) of an ARM, due to a high number of provided inputs (18). In the present pilot study we investigated whether semi-automation might further improve the efficiency of the ITCI, when controlling an ARM. This was achieved by adding a camera to the end effector of the ARM and using computer vision algorithms to guide the ARM to grasp a target object. Three ITCI and one joystick control scheme were tested and compared: 1) manual Cartesian control with a base frame reference point, 2) manual Cartesian control with an end effector reference point 3) manual Cartesian control with an end effector reference point and an autonomous grasp function 4) regular JACO² joystick control. The results indicated that end effector control was superior to the base frame control in total task time, number of commands issued and path efficiency. The addition of the automatic grasp function did not improve the performance, but resulted in fewer collisions/displacements of the target object when grasping.

A review of computer vision for semi-autonomous control of assistive robotic manipulators (ARMs)

Purpose: The advances in artificial intelligence have started to reach a level where autonomous systems are becoming increasingly popular as a way to aid people in their everyday life. Such intelligent systems may especially be beneficially for people struggling to complete common everyday tasks, such as individuals with movement-related disabilities. The focus of this paper is hence to review recent work in using computer vision for semi-autonomous control of assistive robotic manipulators (ARMs). Methods: Four databases were searched using a block search, yielding 257 papers which were reduced to 14 papers after applying various filtering criteria. Each paper was reviewed with focus on the hardware used, the autonomous behaviour achieved using computer vision and the scheme for semi-autonomous control of the system. Each of the reviewed systems were also sought characterized by grading their level of autonomy on a pre-defined scale.Conclusions: A re-occurring issue in the reviewed systems was the inability to handle arbitrary objects. This makes the systems unlikely to perform well outside a controlled environment, such as a lab. This issue could be addressed by having the systems recognize good grasping points or primitive shapes instead of specific pre-defined objects. Most of the reviewed systems did also use a rather simple strategy for the semi-autonomous control, where they switch either between full manual control or full automatic control. An alternative could be a control scheme relying on adaptive blending which could provide a more seamless experience for the user.Implications for rehabilitationAssistive robotic manipulators (ARMs) have the potential to empower individuals with disabilities by enabling them to complete common everyday tasks. This potential can be further enhanced by making the ARM semi-autonomous in order to actively aid the user.The scheme used for the semi-autonomous control of the ARM is crucial as it may be a hindrance if done incorrectly. Especially the ability to customize the semi-autonomous behaviour of the ARM is found to be important.Further research is needed to make the final move from the lab to the homes of the users. Most of the reviewed systems suffer from a rather fixed scheme for the semi-autonomous control and an inability to handle arbitrary objects.

A Confidence-Based Shared Control Strategy for the Smart Tissue Autonomous Robot (STAR)

Autonomous robotic assisted surgery (RAS) systems aim to reduce human errors and improve patient outcomes leveraging robotic accuracy and repeatability during surgical procedures. However, full automation of RAS in complex surgical environments is still not feasible and collaboration with the surgeon is required for safe and effective use. In this work, we utilize our Smart Tissue Autonomous Robot (STAR) to develop and evaluate a shared control strategy for the collaboration of the robot with a human operator in surgical scenarios. We consider 2D pattern cutting tasks with partial blood occlusion of the cutting pattern using a robotic electrocautery tool. For this surgical task and RAS system, we i) develop a confidence-based shared control strategy, ii) assess the pattern tracking performances of manual and autonomous controls and identify the confidence models for human and robot as well as a confidence-based control allocation function, and iii) experimentally evaluate the accuracy of our proposed shared control strategy. In our experiments on porcine fat samples, by combining the best elements of autonomous robot controller with complementary skills of a human operator, our proposed control strategy improved the cutting accuracy by 6.4%, while reducing the operator work time to 44 % compared to a pure manual control.

Recursive Bayesian Human Intent Recognition in Shared-Control Robotics

Effective human-robot collaboration in shared control requires reasoning about the intentions of the human user. In this work, we present a mathematical formulation for human intent recognition during assistive teleoperation under shared autonomy. Our recursive Bayesian filtering approach models and fuses multiple non-verbal observations to probabilistically reason about the intended goal of the user. In addition to contextual observations, we model and incorporate the human agent's behavior as goal-directed actions with adjustable rationality to inform the underlying intent. We examine human inference on robot motion and furthermore validate our approach with a human subjects study that evaluates autonomy intent inference performance under a variety of goal scenarios and tasks, by novice subjects. Results show that our approach outperforms existing solutions and demonstrates that the probabilistic fusion of multiple observations improves intent inference and performance for shared-control operation.

Shared autonomy via hindsight optimization for teleoperation and teaming

In shared autonomy, a user and autonomous system work together to achieve shared goals. To collaborate effectively, the autonomous system must know the user’s goal. As such, most prior works follow a predict-then-act model, first predicting the user’s goal with high confidence, then assisting given that goal. Unfortunately, confidently predicting the user’s goal may not be possible until they have nearly achieved it, causing predict-then-act methods to provide little assistance. However, the system can often provide useful assistance even when confidence for any single goal is low (e.g. move towards multiple goals). In this work, we formalize this insight by modeling shared autonomy as a partially observable Markov decision process (POMDP), providing assistance that minimizes the expected cost-to-go with an unknown goal. As solving this POMDP optimally is intractable, we use hindsight optimization to approximate. We apply our framework to both shared-control teleoperation and human–robot teaming. Compared with predict-then-act methods, our method achieves goals faster, requires less user input, decreases user idling time, and results in fewer user–robot collisions.

Autonomy in Rehabilitation Robotics: An Intersection

Within the field of human rehabilitation, robotic machines are used both to rehabilitate the body and to perform functional tasks. Robotics autonomy able to perceive the external world and reason about high-level control decisions, however, seldom is present in these machines. For functional tasks in particular, autonomy could help to decrease the operational burden on the human and perhaps even to increase access-and this potential only grows as human motor impairments become more severe. There are however serious, and often subtle, considerations to introducing clinically-feasible robotics autonomy to rehabilitation robots and machines. Today the fields of robotics autonomy and rehabilitation robotics are largely separate. The topic of this article is at the intersection of these fields: the introduction of clinically-feasible autonomy solutions to rehabilitation robots, and opportunities for autonomy within the rehabilitation domain.

Soft brain-machine interfaces for assistive robotics: A novel control approach

Robotic systems offer the possibility of improving the life quality of people with severe motor disabilities, enhancing the individual's degree of independence and interaction with the external environment. In this direction, the operator's residual functions must be exploited for the control of the robot movements and the underlying dynamic interaction through intuitive and effective human-robot interfaces. Towards this end, this work aims at exploring the potential of a novel Soft Brain-Machine Interface (BMI), suitable for dynamic execution of remote manipulation tasks for a wide range of patients. The interface is composed of an eye-tracking system, for an intuitive and reliable control of a robotic arm system's trajectories, and a Brain-Computer Interface (BCI) unit, for the control of the robot Cartesian stiffness, which determines the interaction forces between the robot and environment. The latter control is achieved by estimating in real-time a unidimensional index from user's electroencephalographic (EEG) signals, which provides the probability of a neutral or active state. This estimated state is then translated into a stiffness value for the robotic arm, allowing a reliable modulation of the robot's impedance. A preliminary evaluation of this hybrid interface concept provided evidence on the effective execution of tasks with dynamic uncertainties, demonstrating the great potential of this control method in BMI applications for self-service and clinical care.

Head Motion and Head Gesture-Based Robot Control: A Usability Study

The assistive robot system adaptive head motion control for user-friendly support (AMiCUS) has been developed to increase the autonomy of motion impaired people. The six degrees of freedom robot arm with gripper is controlled with head motion and head gestures only, so especially tetraplegics benefit from collaboration with AMiCUS. In this paper, a usability study with a total number of 30 subjects was conducted to validate the AMiCUS interaction technology and design. 24 able-bodied subjects of demographically diverse groups and 6 tetraplegics participated in this paper. All subjects performed different pick and place tasks by controlling AMiCUS. The evaluation of the interaction design was carried out subjectively with a questionnaire as well as objectively by measurement of time, completion rate, and number of trials for correct head gesture performance. The influence of several factors like age, sex, motion impairment, and previous experience on head motion-based human-robot interaction was analyzed. The interaction design has been proven successful in laboratory environment and assessed overall positive by the subjects. The results of the presented paper confirm the usability of the assistive robot AMiCUS. AMiCUS has the potential to benefit tetraplegics by improving their independence in activities of daily living and adapted workplaces.

A decision-theoretic approach for the collaborative control of a smart wheelchair

While assistive robot technology is quickly progressing, several challenges remain to make this technology truly usable and useful for humans. One of the aspects that is particularly important is in defining control protocols that allow both the human and the robot technology to contribute to the best of their abilities. In this paper we propose a framework for the collaborative control of a smart wheelchair designed for individuals with mobility impairments. Our approach is based on a decision-theoretic model of control, and accepts commands from both the human user and robot controller. We use a Partially Observable Markov Decision Process to optimize the collaborative action choice, which allows the system to take into account uncertainty in the user intent, in the command and in the environment. The system is deployed and validated on the SmartWheeler platform, and experiments with 8 users show the improvement in usability and navigation efficiency that are achieved with this form of collaborative control.

Closed-Loop Hybrid Gaze Brain-Machine Interface Based Robotic Arm Control with Augmented Reality Feedback

Brain-machine interface (BMI) can be used to control the robotic arm to assist paralysis people for performing activities of daily living. However, it is still a complex task for the BMI users to control the process of objects grasping and lifting with the robotic arm. It is hard to achieve high efficiency and accuracy even after extensive trainings. One important reason is lacking of sufficient feedback information for the user to perform the closed-loop control. In this study, we proposed a method of augmented reality (AR) guiding assistance to provide the enhanced visual feedback to the user for a closed-loop control with a hybrid Gaze-BMI, which combines the electroencephalography (EEG) signals based BMI and the eye tracking for an intuitive and effective control of the robotic arm. Experiments for the objects manipulation tasks while avoiding the obstacle in the workspace are designed to evaluate the performance of our method for controlling the robotic arm. According to the experimental results obtained from eight subjects, the advantages of the proposed closed-loop system (with AR feedback) over the open-loop system (with visual inspection only) have been verified. The number of trigger commands used for controlling the robotic arm to grasp and lift the objects with AR feedback has reduced significantly and the height gaps of the gripper in the lifting process have decreased more than 50% compared to those trials with normal visual inspection only. The results reveal that the hybrid Gaze-BMI user can benefit from the information provided by the AR interface, improving the efficiency and reducing the cognitive load during the grasping and lifting processes.

An Intention-Driven Semi-autonomous Intelligent Robotic System for Drinking

In this study, an intention-driven semi-autonomous intelligent robotic (ID-SIR) system is designed and developed to assist the severely disabled patients to live independently. The system mainly consists of a non-invasive brain–machine interface (BMI) subsystem, a robot manipulator and a visual detection and localization subsystem. Different from most of the existing systems remotely controlled by joystick, head- or eye tracking, the proposed ID-SIR system directly acquires the intention from users’ brain. Compared with the state-of-art system only working for a specific object in a fixed place, the designed ID-SIR system can grasp any desired object in a random place chosen by a user and deliver it to his/her mouth automatically. As one of the main advantages of the ID-SIR system, the patient is only required to send one intention command for one drinking task and the autonomous robot would finish the rest of specific controlling tasks, which greatly eases the burden on patients. Eight healthy subjects attended our experiment, which contained 10 tasks for each subject. In each task, the proposed ID-SIR system delivered the desired beverage container to the mouth of the subject and then put it back to the original position. The mean accuracy of the eight subjects was 97.5%, which demonstrated the effectiveness of the ID-SIR system.

Design of a wearable interface for lightweight robotic arm for people with mobility impairments

Many common activities of daily living like open a door or fill a glass of water, which most of us take for granted, could be an insuperable problem for people who have limited mobility or impairments. For years the unique alternative to overcame this limitation was asking for human help. Nowadays thanks to recent studies and technology developments, having an assistive devices to compensate the loss of mobility is becoming a real opportunity. Off-the-shelf assistive robotic manipulators have the capability to improve the life of people with motor impairments. Robotic lightweight arms represent one of the most spread solution, in particular some of them are designed specifically to be mounted on wheelchairs to assist users in performing manipulation tasks. On the other hand, usually their control interface relies on joystick and buttons, making the use very challenging for people affected by impaired motor abilities. In this paper, we present a novel wearable control interface for users with limb mobility impairments. We make use of muscles residual motion capabilities, captured through a Body-Machine Interface based on a combination of head tilt estimation and electromyography signals. The proposed BMI is completely wearable, wireless and does not require frequently long calibrations. Preliminary experiments showed the effectiveness of the proposed system for subjects with motor impairments, allowing them to easily control a robotic arm for activities of daily living.

Performing Complex Tasks by Users With Upper-Extremity Disabilities Using a 6-DOF Robotic Arm: A Study

In this paper, we report on the results of a study that was conducted to examine how users suffering from severe upper-extremity disabilities can control a 6 degrees-of-freedom (DOF) robotics arm to complete complex activities of daily living. The focus of the study is not on assessing the robot arm but on examining the human-robot interaction patterns. Three participants were recruited. Each participant was asked to perform three tasks: eating three pieces of pre-cut bread from a plate, drinking three sips of soup from a bowl, and opening a right-handed door with lever handle. Each of these tasks was repeated three times. The arm was mounted on the participant's wheelchair, and the participants were free to move the arm as they wish to complete these tasks. Each task consisted of a sequence of modes where a mode is defined as arm movement in one DOF. Results show that participants used a total of 938 mode movements with an average of 75.5 (std 10.2) modes for the eating task, 70 (std 8.8) modes for the soup task, and 18.7 (std 4.5) modes for the door opening task. Tasks were then segmented into smaller subtasks. It was found that there are patterns of usage per participant and per subtask. These patterns can potentially allow a robot to learn from user's demonstration what is the task being executed and by whom and respond accordingly to reduce user effort.

Human-in-the-Loop Optimization of Shared Autonomy in Assistive Robotics

In this paper, we propose a mathematical framework which formalizes user-driven customization of shared autonomy in assistive robotics as a nonlinear optimization problem. Our insight is to allow the end-user, rather than relying on standard optimization techniques, to perform the optimization procedure, thereby allowing us to leave the exact nature of the cost function indeterminate. We ground our formalism with an interactive optimization procedure that customizes control sharing using an assistive robotic arm. We also present a pilot study that explores interactive optimization with end-users. This study was performed with 17 subjects (4 with spinal cord injury, 13 without injury). Results show all subjects were able to converge to an assistance paradigm, suggesting the existence of optimal solutions. Notably, the amount of assistance was not always optimized for task performance. Instead, some subjects favored retaining more control during the execution over better task performance. The study supports the case for user-driven customization and provides guidance for its continued development and study.

Assistive Teleoperation of Robot Arms via Automatic Time-Optimal Mode Switching

Assistive robotic arms are increasingly enabling users with upper extremity disabilities to perform activities of daily living on their own. However, the increased capability and dexterity of the arms also makes them harder to control with simple, low-dimensional interfaces like joysticks and sip-and-puff interfaces. A common technique to control a high-dimensional system like an arm with a low-dimensional input like a joystick is through switching between multiple control modes. However, our interviews with daily users of the Kinova JACO arm identified mode switching as a key problem, both in terms of time and cognitive load. We further confirmed objectively that mode switching consumes about 17.4% of execution time even for able-bodied users controlling the JACO. Our key insight is that using even a simple model of mode switching, like time optimality, and a simple intervention, like automatically switching modes, significantly improves user satisfaction.

Blending of brain-machine interface and vision-guided autonomous robotics improves neuroprosthetic arm performance during grasping

Background

Recent studies have shown that brain-machine interfaces (BMIs) offer great potential for restoring upper limb function. However, grasping objects is a complicated task and the signals extracted from the brain may not always be capable of driving these movements reliably. Vision-guided robotic assistance is one possible way to improve BMI performance. We describe a method of shared control where the user controls a prosthetic arm using a BMI and receives assistance with positioning the hand when it approaches an object.

Methods

Two human subjects with tetraplegia used a robotic arm to complete object transport tasks with and without shared control. The shared control system was designed to provide a balance between BMI-derived intention and computer assistance. An autonomous robotic grasping system identified and tracked objects and defined stable grasp positions for these objects. The system identified when the user intended to interact with an object based on the BMI-controlled movements of the robotic arm. Using shared control, BMI controlled movements and autonomous grasping commands were blended to ensure secure grasps.

Results

Both subjects were more successful on object transfer tasks when using shared control compared to BMI control alone. Movements made using shared control were more accurate, more efficient, and less difficult. One participant attempted a task with multiple objects and successfully lifted one of two closely spaced objects in 92 % of trials, demonstrating the potential for users to accurately execute their intention while using shared control.

Conclusions

Integration of BMI control with vision-guided robotic assistance led to improved performance on object transfer tasks. Providing assistance while maintaining generalizability will make BMI systems more attractive to potential users.

Trial registration

NCT01364480 and NCT01894802.

Electronic supplementary material

The online version of this article (doi:10.1186/s12984-016-0134-9) contains supplementary material, which is available to authorized users.

High Precision Neural Decoding of Complex Movement Trajectories using Recursive Bayesian Estimation with Dynamic Movement Primitives

Brain-machine interfaces (BMIs) are a rapidly progressing technology with the potential to restore function to victims of severe paralysis via neural control of robotic systems. Great strides have been made in directly mapping a user's cortical activity to control of the individual degrees of freedom of robotic end-effectors. While BMIs have yet to achieve the level of reliability desired for widespread clinical use, environmental sensors (e.g. RGB-D cameras for object detection) and prior knowledge of common movement trajectories hold great potential for improving system performance. Here we present a novel sensor fusion paradigm for BMIs that capitalizes on information able to be extracted from the environment to greatly improve the performance of control. This was accomplished by using dynamic movement primitives to model the 3D endpoint trajectories of manipulating various objects. We then used a switching unscented Kalman filter to continuously arbitrate between the 3D endpoint kinematics predicted by the dynamic movement primitives and control derived from neural signals. We experimentally validated our system by decoding 3D endpoint trajectories executed by a non-human primate manipulating four different objects at various locations. Performance using our system showed a dramatic improvement over using neural signals alone, with median distance between actual and decoded trajectories decreasing from 31.1 cm to 9.9 cm, and mean correlation increasing from 0.80 to 0.98. Our results indicate that our sensor fusion framework can dramatically increase the fidelity of neural prosthetic trajectory decoding.

Assistive Robotic Manipulation through Shared Autonomy and a Body-Machine Interface

Assistive robotic manipulators have the potential to improve the lives of people with motor impairments. They can enable individuals to perform activities such as pick-and-place tasks, opening doors, pushing buttons, and can even provide assistance in personal hygiene and feeding. However, robotic arms often have more degrees of freedom (DoF) than the dimensionality of their control interface, making them challenging to use-especially for those with impaired motor abilities. Our research focuses on enabling the control of high-DoF manipulators to motor-impaired individuals for performing daily tasks. We make use of an individual's residual motion capabilities, captured through a Body-Machine Interface (BMI), to generate control signals for the robotic arm. These low-dimensional controls are then utilized in a shared-control framework that shares control between the human user and robot autonomy. We evaluated the system by conducting a user study in which 6 participants performed 144 trials of a manipulation task using the BMI interface and the proposed shared-control framework. The 100% success rate on task performance demonstrates the effectiveness of the proposed system for individuals with motor impairments to control assistive robotic manipulators.

Integrated vision-based robotic arm interface for operators with upper limb mobility impairments

An integrated, computer vision-based system was developed to operate a commercial wheelchair-mounted robotic manipulator (WMRM). In this paper, a gesture recognition interface system developed specifically for individuals with upper-level spinal cord injuries (SCIs) was combined with object tracking and face recognition systems to be an efficient, hands-free WMRM controller. In this test system, two Kinect cameras were used synergistically to perform a variety of simple object retrieval tasks. One camera was used to interpret the hand gestures to send as commands to control the WMRM and locate the operator's face for object positioning. The other sensor was used to automatically recognize different daily living objects for test subjects to select. The gesture recognition interface incorporated hand detection, tracking and recognition algorithms to obtain a high recognition accuracy of 97.5% for an eight-gesture lexicon. An object recognition module employing Speeded Up Robust Features (SURF) algorithm was performed and recognition results were sent as a command for "coarse positioning" of the robotic arm near the selected daily living object. Automatic face detection was also provided as a shortcut for the subjects to position the objects to the face by using a WMRM. Completion time tasks were conducted to compare manual (gestures only) and semi-manual (gestures, automatic face detection and object recognition) WMRM control modes. The use of automatic face and object detection significantly increased the completion times for retrieving a variety of daily living objects.

Quality of teleoperator adaptive control for telerobotic operations

Extensive studies have been conducted on telerobotic operations for decades due to their widespread applications in a variety of areas. Most studies have been focused on two major issues: stability and telepresence. Few have studied the influence of the operation status of the teleoperator on the performance of telerobotic operations. As subnormal operation status of the teleoperator may result in insufficient and even incorrect operations, the quality of teleoperator (QoT) is an important impact on the performance of the telerobotic operations in terms of the efficiency and safety even if both the stability and telepresence are guaranteed. Therefore, this paper investigates the online identification of the QoT and its application to telerobotic operations. The QoT is identified based on five QoT indicators which are generated based on the teleoperator’s brain EEG signals. A QoT adaptive control method is designed to adapt the velocity and responsivity of the robotic system to the operation status of the teleoperator such that the teleoperation efficiency and safety can be enhanced. The online QoT identification method was conducted on various teleoperators and the QoT adaptive control method was implemented on a mobile manipulator teleoperation system. The experimental results demonstrated the effectiveness and advantages of the proposed methods.

A policy-blending formalism for shared control

In shared control teleoperation, the robot assists the user in accomplishing the desired task, making teleoperation easier and more seamless. Rather than simply executing the user’s input, which is hindered by the inadequacies of the interface, the robot attempts to predict the user’s intent, and assists in accomplishing it. In this work, we are interested in the scientific underpinnings of assistance: we propose an intuitive formalism that captures assistance as policy blending, illustrate how some of the existing techniques for shared control instantiate it, and provide a principled analysis of its main components: prediction of user intent and its arbitration with the user input. We define the prediction problem, with foundations in inverse reinforcement learning, discuss simplifying assumptions that make it tractable, and test these on data from users teleoperating a robotic manipulator. We define the arbitration problem from a control-theoretic perspective, and turn our attention to what users consider good arbitration. We conduct a user study that analyzes the effect of different factors on the performance of assistance, indicating that arbitration should be contextual: it should depend on the robot’s confidence in itself and in the user, and even the particulars of the user. Based on the study, we discuss challenges and opportunities that a robot sharing the control with the user might face: adaptation to the context and the user, legibility of behavior, and the closed loop between prediction and user behavior.