Toward Human-Like Grasp: Functional Grasp by Dexterous Robotic Hand Via Object-Hand Semantic Representation

Intelligent robotic manipulation is a challenging study of machine intelligence. Although many dexterous robotic hands have been designed to assist or replace human hands in executing various tasks, how to teach them to perform dexterous operations like human hands is still a challenge. This motivates us to conduct an in-depth analysis of human behavior in manipulating objects and propose an object-hand manipulation representation. This representation provides an intuitive and clear semantic indication of how the dexterous hand should touch and manipulate an object based on the object's own functional areas. At the same time, we propose a functional grasp synthesis framework, which does not require real grasp label supervision, but relies on the guidance of our object-hand manipulation representation. In addition, in order to obtain better functional grasp synthesis results, we propose a network pre-training method that can make full use of easily obtained stable grasp data, and a network training strategy to coordinate the loss functions. We conduct object manipulation experiments on a real robot platform, and evaluate the performance and generalization of our object-hand manipulation representation and grasp synthesis framework.

Sequential contact-based adaptive grasping for robotic hands

This paper proposes a novel type of grasping strategy that draws inspiration from the role of touch and the importance of wrist motions in human grasping. The proposed algorithm, which we call Sequential Contact-based Adaptive Grasping, can be used to reactively modify a given grasp plan according to contacts arising between the hand and the object. This technique, based on a systematic constraint categorization and an iterative task inversion procedure, is shown to lead to synchronized motions of the fingers and the wrist, as it can be observed in humans, and to increase grasp success rate by substantially mitigating the relevant problems of object slippage during hand closure and of uncertainties caused by the environment and by the perception system. After describing the grasping problem in its quasi-static aspects, the algorithm is derived and discussed with some simple simulations. The proposed method is general as it can be applied to different kinds of robotic hands. It refines a priori defined grasp plans and significantly reduces their accuracy requirements by relying only on a forward kinematic model and elementary contact information. The efficacy of our approach is confirmed by experimental results of tests performed on a collaborative robot manipulator equipped with a state-of-the-art underactuated soft hand.

An inverse optimization approach to understand human acquisition of kinematic coordination in bimanual fine manipulation tasks

Tasks that require the cooperation of both hands and arms are common in human everyday life. Coordination helps to synchronize in space and temporally motion of the upper limbs. In fine bimanual tasks, coordination enables also to achieve higher degrees of precision that could be obtained from a single hand. We studied the acquisition of bimanual fine manipulation skills in watchmaking tasks, which require assembly of pieces at millimeter scale. It demands years of training. We contrasted motion kinematics performed by novice apprentices to those of professionals. Fifteen subjects, ten novices and five experts, participated in the study. We recorded force applied on the watch face and kinematics of fingers and arms. Results indicate that expert subjects wisely place their fingers on the tools to achieve higher dexterity. Compared to novices, experts also tend to align task-demanded force application with the optimal force transmission direction of the dominant arm. To understand the cognitive processes underpinning the different coordination patterns across experts and novice subjects, we followed the optimal control theoretical framework and hypothesize that the difference in task performances is caused by changes in the central nervous system's optimal criteria. We formulated kinematic metrics to evaluate the coordination patterns and exploit inverse optimization approach to infer the optimal criteria. We interpret the human acquisition of novel coordination patterns as an alteration in the composition structure of the central nervous system's optimal criteria accompanied by the learning process.

Adaptive Neural Network Tracking Control for Robotic Manipulators With Dead Zone

In this paper, the adaptive neural network (NN) tracking control problem is addressed for robot manipulators subject to dead-zone input. The control objective is to design an adaptive NN controller to guarantee the stability of the systems and obtain good performance. Different from the existing results, which used NN to approximate the nonlinearities directly, NNs are employed to identify the originally designed virtual control signals with unknown nonlinear items in this paper. Moreover, a sequence of virtual control signals and real controller are designed. The adaptive backstepping control method and Lyapunov stability theory are used to prove the proposed controller can ensure all the signals in the systems are semiglobally uniformly ultimately bounded, and the output of the systems can track the reference signal closely. Finally, the proposed adaptive control strategy is applied to the Puma 560 robot manipulator to demonstrate its effectiveness.

Description and Analysis of Multi-Fingered Hand Grasping with a New Finger-Object Contact Model

The proper description of finger-object contact state is critical to the analysis and control of multi-fingered hand grasping. At present, most studies assume that the finger-object contact state is invariable. However, in different stages of hand grasping, finger-object contact state is usually variable due to different contact factors, therefore, current finger-object contact models are not very suitable for describing it and the existing description and analysis of the hand grasping may be not appropriate. In this paper, we proposed a new method for describing and analyzing the multi-fingered hand grasping under various finger-object contact states, including a new finger-object contact model for describing the actual finger-object contact state, a grasp matrix for describing the hand grasping, and the condition number of the grasp matrix for analyzing the hand grasping. Taking a three-fingered hand grasping a sphere as an example, the hand grasping under four types of contact states was investigated. Simulations and experimental results both validated the effectiveness of the proposed method.

Learning task-oriented grasping for tool manipulation from simulated self-supervision

Tool manipulation is vital for facilitating robots to complete challenging task goals. It requires reasoning about the desired effect of the task and, thus, properly grasping and manipulating the tool to achieve the task. Most work in robotics has focused on task-agnostic grasping, which optimizes for only grasp robustness without considering the subsequent manipulation tasks. In this article, we propose the Task-Oriented Grasping Network (TOG-Net) to jointly optimize both task-oriented grasping of a tool and the manipulation policy for that tool. The training process of the model is based on large-scale simulated self-supervision with procedurally generated tool objects. We perform both simulated and real-world experiments on two tool-based manipulation tasks: sweeping and hammering. Our model achieves overall 71.1% task success rate for sweeping and 80.0% task success rate for hammering.

Realtime Hand-Object Interaction Using Learned Grasp Space for Virtual Environments

We present a realtime virtual grasping algorithm to model interactions with virtual objects. Our approach is designed for multi-fingered hands and makes no assumptions about the motion of the user's hand or the virtual objects. Given a model of the virtual hand, we use machine learning and particle swarm optimization to automatically pre-compute stable grasp configurations for that object. The learning pre-computation step is accelerated using GPU parallelization. At runtime, we rely on the pre-computed stable grasp configurations, and dynamics/non-penetration constraints along with motion planning techniques to compute plausible looking grasps. In practice, our realtime algorithm can perform virtual grasping operations in less than 20ms for complex virtual objects, including high genus objects with holes. We have integrated our grasping algorithm with Oculus Rift HMD and Leap Motion controller and evaluated its performance for different tasks corresponding to grabbing virtual objects and placing them at arbitrary locations. Our user evaluation suggests that our virtual grasping algorithm can increase the user's realism and participation in these tasks and offers considerable benefits over prior interaction algorithms, such as pinch grasping and raycast picking.

Evaluation of Human Prehension Using Grasp Quality Measures

One of the main features of the human hand is its grasping ability. Robot grasping has been studied for years and different quality measures have been proposed to evaluate the stability and manipulability of grasps. Although the human hand is obviously more complex than robot hands, the methods used in robotics might be adopted to study the human grasp. The purpose of this work is to propose a set of measures that allow the evaluation of different aspects of the human grasp. The most common robotic grasp quality measures have been adapted to the evaluation of the human hand and a new quality measure – the fatigue index – is proposed in order to incorporate the biomechanical aspect into the evaluation. The minimum set of indices that allows the evaluation of the different aspects of the grasp is obtained from the analysis of a human prehension experiment.

Optimal grasp planning for a dexterous robotic hand using the volume of a generalized force ellipsoid during accepted flattening

A grasp planning method based on the volume and flattening of a generalized force ellipsoid is proposed to improve the grasping ability of a dexterous robotic hand. First, according to the general solution of joint torques for a dexterous robotic hand, a grasping indicator for the dexterous hand—the maximum volume of a generalized external force ellipsoid and the minimum volume of a generalized contact internal force ellipsoid during accepted flattening—is proposed. Second, an optimal grasp planning method based on a task is established using the grasping indicator as an objective function. Finally, a simulation analysis and grasping experiment are performed. Results show that when the grasping experiment is conducted with the grasping configuration and positions of contact points optimized using the proposed grasping indicator, the root-mean-square values of the joint torques and contact internal forces of the dexterous hand are at a minimum. The effectiveness of the proposed grasping planning method is thus demonstrated.

On redundancy resolution of the human thumb, index and middle fingers in cooperative object translation

Redundancy in motion, and synergy in neuromuscular coordination provides significant versatility to the human fingers while performing coordinated grasping and manipulation tasks in several ways. This paper explores how humans may resolve the redundancy in their thumb, index and middle fingers when these digits flex to cooperatively translate a small object toward the palm. It is observed that humans actively employ a secondary subtask of maximizing instantaneous manipulability that helps determine all intermediate finger configurations when performing the primary subtask of following a tip trajectory. This behavior is accurately captured by an inverse kinematic model based on aredundancyparameter. The joint angles get determined unambiguously though the redundancy parameter is shown to depend on the instantaneous finger configurations and also, to attain negative values. Further, this parameter is noted to vary significantly across subjects performing the same kinematic task. The findings, that are based on the experimental finger motion data garnered from 12 subjects, are reckoned to be of significant importance, especially in reference to the challenges in design and control of finger exoskeletons for cooperative manipulation.

Coping with the Grasping Uncertainties in Force-closure Analysis

Friction uncertainty and contact position uncertainty may have a disastrous effect on the closure properties of grasps. This paper reflects our approach to handling these uncertainties in force-closure analysis. The former uncertainty is measured by the possible reduction rate κ of friction coefficients, while the radius ρ of contact regions is used to quantify the latter uncertainty. The actual contact point may deviate from the desired position but not farther than ρ ρS, the supremum of ρ without loss of force-closure, indicates the grasp tolerance to contact position uncertainty. For investigating the above uncertainties systematically, we propose three new problems in force-closure: whether a grasp with given κ and ρ achieves force-closure, what value ρS equals if κ is given, and how ρS varies versus κ. To facilitate their solutions, we extend the scope of the infinitesimal motion approach from form-closure analysis to force-closure. A necessary and sufficient condition for force-closure is deduced by means of the duality between some convex cones, which play the key role in solving the problems. Finally, efficient algorithms are developed and applied to two illustrative examples.

Robot Grasp Synthesis Algorithms: A Survey

This article presents a survey of the existing computational algorithms meant for achieving four important properties in autonomous multifingered robotic hands. The four properties are: dexterity, equilibrium, stability, and dynamic behavior The multifingered robotic hands must be controlled so as to possess these properties and hence be able to autonomously perform complex tasks in a way similar to human hands.

Existing algorithms to achieve dexterity primarily involve solving an unconstrained linear programming problem where an objective function can be chosen to represent one or more of the currently known dexterity measures. Algorithms to achieve equilibrium also constitute solving a linear program ming problem wherein the positivity, friction, and joint torque constraints of all fingers are accounted for while optimizing the internal grasping forces. Stability algorithms aim at achiev ing positive definite grasp impedance matrices by solving for the required fingertip impedances. This problem reduces to a nonlinear programming problem. Dynamic behavior algorithms determine fingertip impedances, which, when achieved, lead to a desired dynamic behavior. This problem, too, becomes a linear programming problem.

If a robotic hand has to acquire any or all of these proper ties, the corresponding algorithms should become integral parts of the hand control system. These algorithms are collectively referred to in this article as robot grasp synthesis algorithms.

A Compact Inverse Velocity Solution for Redundant Robots

A compact inverse rate kinematics solution for serial-chain, redundant, n-jointed robots is presented. The serial-chain inverse rate kinematics problem is dual to the inverse statics problem in in-parallel systems such as cooperating robots, in which the task of distributing the load wrench between multiple subsystems is typically underspecified. Therefore, methods for obtaining the wrench distribution in an in-parallel system can be applied to the underconstrained inverse kinematics problem for serial-chain manipulators. In particular, a solution that minimizes a weighted-norm of the joint rates vector is derived in this article. The advantages of this method are the ability to accommodate any type of joint and the availability of analytical, closed-form solutions for the joint rates. Finally, a geometric interpretation of the solution is provided using screw system theory.

On Computing the Friction Forces Associated With Three-Fingered Grasp

Computation of friction forces between fingers and object for a given external force is important in grasp planning. Based on the required friction forces, the magnitudes of the grasping forces can be adjusted to avoid slippage of the object be tween fingers. This article presents a very elegant and effective method for computing the friction forces between the fingertips of a dexterous mechanical hand and the object as a function of the external wrench which can act on the object. Fingertips are modeled such that they are able to exert normal forces and are making point contact with friction with the object. The method in this article utilizes geometric information on the grasp con figuration, screw geometry, and bi-orthogonal systems. The method is demonstrated through detailed tutorial solutions of two examples.

A soft-contact and wrench based approach to study grasp planning and execution

Grasping research in robotics has made remarkable progress in the last three decades and sophisticated computational tools are now available for planning robotic grasping in complex environments. However, studying the neural control of prehension in humans is more complex than studying robotic grasping. The elaborate musculoskeletal geometries and complex neural inputs to the hand facilitate a symphonic interplay of power and precision that allows humans to grasp fragile objects in a stable way without either crushing or dropping them. Most prehension studies have focused on a planar simplification of prehension since planar analyses render the complex problem of prehension tractable with few variables. The caveat is that planar simplification allows researchers to ask only a limited set of questions. In fact, one of the problems with extending prehension studies to three dimensions is the lack of analytical tools for quantifying features of spatial prehension. The current paper provides a theoretical adaptation and a step-by-step implementation of a widely used soft-contact wrench model for spatial human prehension. We propose two indices, grasp caliber and grasp intensity, to quantitatively relate digit placement and digit forces to grasp stability. Grasp caliber is the smallest singular value of the grasp matrix and it indicates the proximity of the current grasp configuration to instability. Grasp intensity is the magnitude of the excessive wrench applied by the digits to counter perturbations. Apart from quantifying stability of spatial grasps, these indices can also be applied to investigate sensory-motor coupling and the role of perception in grasp planning.

Grasp planning to maximize task coverage

This paper proposes a task-oriented grasp quality metric based on distribution of task disturbance, which could be used to search for a grasp that covers the most significant part of the disturbance distribution. Rather than using a uniformly distributed task wrench space, this paper models a manipulation task with a non-parametric statistical distribution model built from the disturbance data captured during the task demonstrations. The grasp resulting from maximizing the proposed grasp quality criterion is prone to increasing the coverage of most frequent disturbances. To reduce the computational complexity of the search in a high-dimensional robotic hand configuration space, as well as to avoid the correspondence problem, the candidate grasps are computed from a reduced configuration space that is confined by a set of given thumb placements and thumb directions. The proposed approach has been tested both in simulation and on a real robotic system. In simulation, the approach was validated with a Barrett hand and a Shadow hand in several manipulation tasks. Experiments on a physical robotic platform verified the consistency between the proposed grasp metric and the success rate.

Learning grasps with topographic features

We present a system for grasping unknown objects, even from piles or cluttered scenes, given a point cloud. Our method is based on the topography of a given scene and abstracts grasp-relevant structures to enable machine learning techniques for grasping tasks. We describe how Height Accumulated Features (HAF) and their extension, Symmetry Height Accumulated Features, extract grasp relevant local shapes. We investigate grasp quality using an F-score metric. We demonstrate the gain and the expressive power of HAF by comparing its trained classifier with one that resulted from training on simple height grids. An efficient way to calculate HAF is presented. We describe how the trained grasp classifier is used to explore the whole grasp space and introduce a heuristic to find the most robust grasp. We show how to use our approach to adapt the gripper opening width before grasping. In robotic experiments we demonstrate different aspects of our system on three robot platforms: a Schunk seven-degree-of-freedom arm, a PR2 and a Kuka LWR arm. We perform tasks to grasp single objects, autonomously unload a box and clear the table. Thereby we show that our approach is easily adaptable and robust with respect to different manipulators. As part of the experiments we compare our algorithm with a state-of-the-art method and show significant improvements. Concrete examples are used to illustrate the benefit of our approach compared with established grasp approaches. Finally, we show advantages of the symbiosis between our approach and object recognition.

Grasp quality measures: review and performance

The correct grasp of objects is a key aspect for the right fulfillment of a given task. Obtaining a good grasp requires algorithms to automatically determine proper contact points on the object as well as proper hand configurations, especially when dexterous manipulation is desired, and the quantification of a good grasp requires the definition of suitable grasp quality measures. This article reviews the quality measures proposed in the literature to evaluate grasp quality. The quality measures are classified into two groups according to the main aspect they evaluate: location of contact points on the object and hand configuration. The approaches that combine different measures from the two previous groups to obtain a global quality measure are also reviewed, as well as some measures related to human hand studies and grasp performance. Several examples are presented to illustrate and compare the performance of the reviewed measures.

Grasping deformable planar objects: Squeeze, stick/slip analysis, and energy-based optimalities

Robotic grasping of deformable objects is difficult and under-researched, not simply due to the high computational cost of modeling. More fundamentally, several issues arise with the deformation of an object being grasped: a changing wrench space, growing finger contact areas, and pointwise varying contact modes inside these areas. Consequently, contact constraints needed for deformable modeling are hardly established at the beginning of the grasping operation. This paper presents a grasping strategy that squeezes the object with two fingers under specified displacements rather than forces. A ‘stable’ squeeze minimizes the potential energy for the same amount of squeezing, while a ‘pure’ squeeze ensures that the object undergoes no rigid body motion as it deforms. Assuming linear elasticity, a finite element analysis guarantees equilibrium and the uniqueness of deformation during a squeeze action. An event-driven algorithm tracks the contact regions as well as the modes of contact in their interiors under Coulomb friction, which in turn serve as the needed constraints for deformation update. Grasp quality is characterized as the amount of work performed by the grasping fingers in resisting a known push by some adversary finger. Simulation and multiple experiments have been conducted to validate the results over solid and ring-like 2D objects.

Preoperative analysis of the stability of fit of a patient-specific surgical guide

Although the use of patient-specific surgical guides has gained popularity over the past decade, little research has been done to examine in an objective and qualitative way the fit of such instruments. In this study, we have developed a model to predict the stability of a guide designed to fit on a supporting bone surface, thereby providing feedback on the translational and rotational stability of the device. The method was validated by comparing different guide designs with respect to their stability on the contact surface and comparing these results to those measured with a set of experiments. This validation experiment indicates that our stability model can be used to predict the stability of the fit of a surgical guide during the preoperative design process.

A general dynamic force distribution algorithm for multifingered grasping

A general dynamic force distribution (DFD) algorithm for multifingered grasping is presented. Based on the convexity and cone property of contact constraints, the optimal internal force is defined. Determination of the contact force is divided into two phases: (1) determine the direction of the optimal internal force during grasp planning and (2) determine the manipulation force and compensation factor during task manipulation. The total computation cost consists of off-line numerical iterations and on-line analytical computations. Only the latter is related to the real-time control. The algorithm can directly deal with the nonlinear and/or coupled constraints and well overcome the temporal discontinuity.

A Novel Synthesis of Computational Approaches Enables Optimization of Grasp Quality of Tendon-Driven Hands

We propose a complete methodology to find the full set of feasible grasp wrenches and the corresponding wrench-direction-independent grasp quality for a tendon-driven hand with arbitrary design parameters. Monte Carlo simulations on two representative designs combined with multiple linear regression identified the parameters with the greatest potential to increase this grasp metric. This synthesis of computational approaches now enables the systematic design, evaluation, and optimization of tendon-driven hands.

Evaluation of 3D grasps with physical interpretations using object wrench space

In this paper we propose an intuitive and practical grasp quality measure for grasping 3D objects with a multi-fingered robot hand. The proposed measure takes into account the object geometries through the concept of object wrench space. Physically, the positive measure value has a meaning of the minimum single disturbance that grasp cannot resist, while the negative measure value implies the minimum necessary helping force that restores a non-force-closure grasp into a force-closure one. We show that the measure value is invariant between similar grasps and also between different torque origins. We verify the validity of the proposed measure via simulations by using computer models of a three-fingered robot hand and polygonal objects.

Task-Driven Extraction of Object Contour by Human Haptics: Part 2

The extraction of contour information from subjects is essential for purposes of grasping and manipulation. We proposed that human haptic exploration of contours, in the absence of vision, would reveal specialized patterns, or “contour exploration procedures,” that are directly related to task goals and intrinsic system capacities. Our general assumptions, method, and initial results were described in Part 1. Part 2 provides an analysis of the relation between contour extraction procedures and processing constraints. These theoretical assumptions are supported by empirical findings, and implications are discussed for issues of importance to robotic exploration and manipulation.

Geometric modelling of bounded and frictional grasps

This paper, as an extension of an earlier paper, presents a geometrical representation and theoretical foundation of robot grasping that is affected by friction and by the magnitudes of normal contact forces. Grasp analysis and synthesis are based on the concepts of constraint cone (region), restraint cone and freedom cone in the force screw-space and location screw-space. In fact, the freedom cone is the aggregate of all the screws repelling and reciprocal to the grasp geometry (denoted by the restraint cone). Moreover, the constraint cone and the constraint region describe the effect of friction and the influence of the amplitudes of normal forces. The conditions of equilibrium grasp, stable grasp, form closure, force closure, and relative form closure are derived from the geometrical representation, and approaches are suggested for the design of adaptable fixtures, and for automatic grasp planning. It is shown that relative form closure is of great significance for frictional grasp, since it is possible to generate enough internal forces in the subspace spanned by the contacts and consequently produce corresponding friction forces in the reciprocal subspace.

Geometric modelling of boundless grasps

This paper describes the development of the topological and geometric approach to the analysis and synthesis of form-closed grasps of arbitrary objects. Concepts of wrench spaces and the relevant subsets of the positive cone, the affine hull, the convex huli, and the related polar cone are used, and applied to a representative example.

Inference on robotic assembly precedence constraints using a part contact level graph

This paper describes a new approach to the automatic generation of assembly precedence constraints for robotic assembly, using a part contact level graph. Since inference of precedence constraints is a prerequisite to generate assembly sequences of a product, much work has been done in this field. However, most of it has some limitations in that they use a cumbersome user query or time-consuming geometric reasoning. To cope with these problems, this paper utilizes three directional part contact level graphs which, in three orthogonal directions, contain the information on directional connections for each pair of mating parts. By using these graphs, an assembly precedence constraint is inferred in two steps: The first step infers a precedence constraint for each directional connection by applying the path-finding algorithm. Utilizing the precedence constraints thus obtained, the next step infers the precedence constraint for each part to be assembled with its base assembly. Examples are given to illustrate the concepts and procedure of the proposed scheme.

Grasping, coordination and optimal force distribution in multifingered mechanisms

In the field of multifingered mechanisms the control/command problem is mainly a problem o1 coordination. The problem is not only to coordinate joints of a chains but also to coordinate the different chains together.

This paper presents a general and efficient method for implementing the control/command of such systems, taking into account the force distribution problem. To solve this problem it is necessary to pay great attention to dynamic effects. To do this, we broke down the Inverse Dynamic Model (I.D.M.) problem into two main levels; One level is devoted to I.D.M. computation; it can be called the Finger Level (F.L.). As we wanted to divide up the work to be done as much as possible, we subdivided the Finger Level according to the number o1 kinematic chains. In addition, we considered a second level, the Coordinator. This level has to control all the chains using the Fingers-to-Object-Interaction Model (F.O.LM.).

Next, we will also introduce new grasping systems: Polyvalent Gripper Systems (P.G.S). There are a new solution to multicomponent assembly problems. As they can be equipped with several multifingered mechanisms, they can also use the control/command scheme.