Nonprehensile Manipulation for Rapid Object Spinning via Multisensory Learning from Demonstration

Dexterous manipulation concerns the control of a robot hand to manipulate an object in a desired manner. While classical dexterous manipulation strategies are based on stable grasping (or force closure), many human-like manipulation tasks do not maintain grasp stability and often utilize the dynamics of the object rather than the closed form of kinematic relation between the object and the robotic hand. Such manipulation strategies are referred as nonprehensile or dynamic dexterous manipulation in the literature. Nonprehensile manipulation often involves fast and agile movements such as throwing and flipping. Due to the complexity of such motions and uncertainties associated with them, it has been challenging to realize nonprehensile manipulation tasks in a reliable way. In this paper, we propose a new control strategy to realize practical nonprehensile manipulation. First, we make explicit use of multiple modalities of sensory data for the design of control law. Specifically, force data are employed for feedforward control, while position data are used for feedback control. Secondly, control signals (both feedback and feedforward) are obtained through multisensory learning from demonstration (LfD) experiments designed and performed for specific nonprehensile manipulation tasks of concern. To prove the concept of the proposed control strategy, experimental tests were conducted for a dynamic spinning task using a sensory-rich, two-finger robotic hand. The control performance (i.e., the speed and accuracy of the spinning task) was also compared with that of classical dexterous manipulation based on force closure and finger gaiting.

Distinct adaptation processes underlie multidigit force coordination for dexterous manipulation

The human sensorimotor system can adapt to various changes in the environmental dynamics by updating motor commands to improve performance after repeated exposure to the same task. However, the characteristics and mechanisms of the adaptation process remain unknown for dexterous manipulation, a unique motor task in which the body physically interacts with the environment with multiple effectors, i.e., digits, in parallel. We addressed this gap by using robotic manipulanda to investigate the changes in the digit force coordination following mechanical perturbation of an object held by tripod grasps. As the participants gradually adapted to lifting the object under perturbations, we quantified two components of digit force coordination. One is the direction-specific manipulation moment that directly counteracts the perturbation, whereas the other one is the direction-independent internal moment that supports the stability and stiffness of the grasp. We found that trial-to-trial improvement of task performance was associated with increased manipulation moment and a gradual decrease of the internal moment. These two moments were characterized by different rates of adaptation. We also examined how these two force coordination components respond to changes in perturbation directions. Importantly, we found that the manipulation moment was sensitive to the extent of repetitive exposure to the previous context that has an opposite perturbation direction, whereas the internal moment did not. However, the internal moment was sensitive to whether the postchange perturbation direction was previously experienced. Our results reveal, for the first time, that two distinct processes underlie the adaptation of multidigit force coordination for dexterous manipulation.NEW & NOTEWORTHY Changes in digit force coordination in multidigit object manipulation were quantified with a novel experimental design in which human participants adapted to mechanical perturbations applied to the object. Our results show that the adaptation of digit force coordination can be characterized by two distinct components that operate at different timescales. We further show that these two components respond to changes in perturbation direction differently.

An Accessible, Open-Source Dexterity Test: Evaluating the Grasping and Dexterous Manipulation Capabilities of Humans and Robots

Evaluating the dexterity of human and robotic hands through appropriate benchmarks, scores, and metrics is of paramount importance for determining how skillful humans are and for designing and developing new bioinspired or even biomimetic end-effectors (e.g., robotic grippers and hands). Dexterity tests have been used in industrial and medical settings to assess how dexterous the hands of workers and surgeons are as well as in robotic rehabilitation settings to determine the improvement or deterioration of the hand function after a stroke or a surgery. In robotics, having a comprehensive dexterity test can allow us to evaluate and compare grippers and hands irrespectively of their design characteristics. However, there is a lack of well defined metrics, benchmarks, and tests that quantify robot dexterity. Previous work has focused on a number of widely accepted functional tests that are used for the evaluation of manual dexterity and human hand function improvement post injury. Each of these tests focuses on a different set of specific tasks and objects. Deriving from these tests, this work proposes a new modular, affordable, accessible, open-source dexterity test for both humans and robots. This test evaluates the grasping and manipulation capabilities by combining the features and best practices of the aforementioned tests, as well as new task categories specifically designed to evaluate dexterous manipulation capabilities. The dexterity test and the accompanying benchmarks allow us to determine the overall hand function recovery and dexterity of robotic end-effectors with ease. More precisely, a dexterity score that ranges from 0 (simplistic, non-dexterous system) to 1 (human-like system) is calculated using the weighted sum of the accuracy and task execution speed subscores. It should also be noted that the dexterity of a robotic system can be evaluated assessing the efficiency of either the robotic hardware, or the robotic perception system, or both. The test and the benchmarks proposed in the study have been validated using extensive human and robot trials. The human trials have been used to determine the baseline scores for the evaluation system. The results show that the time required to complete the tasks reduces significantly with trials indicating a clear learning curve in mastering the dexterous manipulation capabilities associated with the imposed tasks. Finally, the time required to complete the tasks with restricted tactile feedback is significantly higher indicating its importance.

Tactile Sensors for Parallel Grippers: Design and Characterization

Tactile data perception is of paramount importance in today’s robotics applications. This paper describes the latest design of the tactile sensor developed in our laboratory. Both the hardware and firmware concepts are reported in detail in order to allow the research community the sensor reproduction, also according to their needs. The sensor is based on optoelectronic technology and the pad shape can be adapted to various robotics applications. A flat surface, as the one proposed in this paper, can be well exploited if the object sizes are smaller than the pad and/or the shape recognition is needed, while a domed pad can be used to manipulate bigger objects. Compared to the previous version, the novel tactile sensor has a larger sensing area and a more robust electronic, mechanical and software design that yields less noise and higher flexibility. The proposed design exploits standard PCB manufacturing processes and advanced but now commercial 3D printing processes for the realization of all components. A GitHub repository has been prepared with all files needed to allow the reproduction of the sensor for the interested reader. The whole sensor has been tested with a maximum load equal to 15N, by showing a sensitivity equal to 0.018V/N. Moreover, a complete and detailed characterization for the single taxel and the whole pad is reported to show the potentialities of the sensor also in terms of response time, repeatability, hysteresis and signal to noise ratio.

Learning for a Robot: Deep Reinforcement Learning, Imitation Learning, Transfer Learning

Dexterous manipulation of the robot is an important part of realizing intelligence, but manipulators can only perform simple tasks such as sorting and packing in a structured environment. In view of the existing problem, this paper presents a state-of-the-art survey on an intelligent robot with the capability of autonomous deciding and learning. The paper first reviews the main achievements and research of the robot, which were mainly based on the breakthrough of automatic control and hardware in mechanics. With the evolution of artificial intelligence, many pieces of research have made further progresses in adaptive and robust control. The survey reveals that the latest research in deep learning and reinforcement learning has paved the way for highly complex tasks to be performed by robots. Furthermore, deep reinforcement learning, imitation learning, and transfer learning in robot control are discussed in detail. Finally, major achievements based on these methods are summarized and analyzed thoroughly, and future research challenges are proposed.

Hands in the Real World

Robots face a rapidly expanding range of potential applications beyond controlled environments, from remote exploration and search-and-rescue to household assistance and agriculture. The focus of physical interaction is typically delegated to end-effectors-fixtures, grippers or hands-as these machines perform manual tasks. Yet, effective deployment of versatile robot hands in the real world is still limited to few examples, despite decades of dedicated research. In this paper we review hands that found application in the field, aiming to discuss open challenges with more articulated designs, discussing novel trends and perspectives. We hope to encourage swift development of capable robotic hands for long-term use in varied real world settings. The first part of the paper centers around progress in artificial hand design, identifying key functions for a variety of environments. The final part focuses on the overall trends in hand mechanics, sensors and control, and how performance and resiliency are qualified for real world deployment.

Design and Calibration of a Force/Tactile Sensor for Dexterous Manipulation

This paper presents the design and calibration of a new force/tactile sensor for robotic applications. The sensor is suitably designed to provide the robotic grasping device with a sensory system mimicking the human sense of touch, namely, a device sensitive to contact forces, object slip and object geometry. This type of perception information is of paramount importance not only in dexterous manipulation but even in simple grasping tasks, especially when objects are fragile, such that only a minimum amount of grasping force can be applied to hold the object without damaging it. Moreover, sensing only forces and not moments can be very limiting to securely grasp an object when it is grasped far from its center of gravity. Therefore, the perception of torsional moments is a key requirement of the designed sensor. Furthermore, the sensor is also the mechanical interface between the gripper and the manipulated object, therefore its design should consider also the requirements for a correct holding of the object. The most relevant of such requirements is the necessity to hold a torsional moment, therefore a soft distributed contact is necessary. The presence of a soft contact poses a number of challenges in the calibration of the sensor, and that is another contribution of this work. Experimental validation is provided in real grasping tasks with two sensors mounted on an industrial gripper.

The TacTip Family: Soft Optical Tactile Sensors with 3D-Printed Biomimetic Morphologies

Tactile sensing is an essential component in human-robot interaction and object manipulation. Soft sensors allow for safe interaction and improved gripping performance. Here we present the TacTip family of sensors: a range of soft optical tactile sensors with various morphologies fabricated through dual-material 3D printing. All of these sensors are inspired by the same biomimetic design principle: transducing deformation of the sensing surface via movement of pins analogous to the function of intermediate ridges within the human fingertip. The performance of the TacTip, TacTip-GR2, TacTip-M2, and TacCylinder sensors is here evaluated and shown to attain submillimeter accuracy on a rolling cylinder task, representing greater than 10-fold super-resolved acuity. A version of the TacTip sensor has also been open-sourced, enabling other laboratories to adopt it as a platform for tactile sensing and manipulation research. These sensors are suitable for real-world applications in tactile perception, exploration, and manipulation, and will enable further research and innovation in the field of soft tactile sensing.

Fusion of Haptic and Gesture Sensors for Rehabilitation of Bimanual Coordination and Dexterous Manipulation

Disabilities after neural injury, such as stroke, bring tremendous burden to patients, families and society. Besides the conventional constrained-induced training with a paretic arm, bilateral rehabilitation training involves both the ipsilateral and contralateral sides of the neural injury, fitting well with the fact that both arms are needed in common activities of daily living (ADLs), and can promote good functional recovery. In this work, the fusion of a gesture sensor and a haptic sensor with force feedback capabilities has enabled a bilateral rehabilitation training therapy. The Leap Motion gesture sensor detects the motion of the healthy hand, and the omega.7 device can detect and assist the paretic hand, according to the designed cooperative task paradigm, as much as needed, with active force feedback to accomplish the manipulation task. A virtual scenario has been built up, and the motion and force data facilitate instantaneous visual and audio feedback, as well as further analysis of the functional capabilities of the patient. This task-oriented bimanual training paradigm recruits the sensory, motor and cognitive aspects of the patient into one loop, encourages the active involvement of the patients into rehabilitation training, strengthens the cooperation of both the healthy and impaired hands, challenges the dexterous manipulation capability of the paretic hand, suits easy of use at home or centralized institutions and, thus, promises effective potentials for rehabilitation training.

Vision-Based Control of a Handheld Surgical Micromanipulator with Virtual Fixtures

Performing micromanipulation and delicate operations in submillimeter workspaces is difficult because of destabilizing tremor and imprecise targeting. Accurate micromanipulation is especially important for microsurgical procedures, such as vitreoretinal surgery, to maximize successful outcomes and minimize collateral damage. Robotic aid combined with filtering techniques that suppress tremor frequency bands increases performance; however, if knowledge of the operator's goals is available, virtual fixtures have been shown to further improve performance. In this paper, we derive a virtual fixture framework for active handheld micromanipulators that is based on high-bandwidth position measurements rather than forces applied to a robot handle. For applicability in surgical environments, the fixtures are generated in real-time from microscope video during the procedure. Additionally, we develop motion scaling behavior around virtual fixtures as a simple and direct extension to the proposed framework. We demonstrate that virtual fixtures significantly outperform tremor cancellation algorithms on a set of synthetic tracing tasks (p < 0.05). In more medically relevant experiments of vein tracing and membrane peeling in eye phantoms, virtual fixtures can significantly reduce both positioning error and forces applied to tissue (p < 0.05).