Human hand descriptions and gesture recognition for object manipulation

Design of a Wearable Real-Time Hand Motion Tracking System Using an Array of Soft Polymer Acoustic Waveguides

Robust hand motion tracking holds promise for improved human-machine interaction in diverse fields, including virtual reality, and automated sign language translation. However, current wearable hand motion tracking approaches are typically limited in detection performance, wearability, and durability. This article presents a hand motion tracking system using multiple soft polymer acoustic waveguides (SPAWs). The innovative use of SPAWs as strain sensors offers several advantages that address the limitations. SPAWs are easily manufactured by casting a soft polymer shaped as a soft acoustic waveguide and containing a commercially available small ceramic piezoelectric transducer. When used as strain sensors, SPAWs demonstrate high stretchability (up to 100%), high linearity (R2 > 0.996 in all quasi-static, dynamic, and durability tensile tests), negligible hysteresis (<0.7410% under strain of up to 100%), excellent repeatability, and outstanding durability (up to 100,000 cycles). SPAWs also show high accuracy for continuous finger angle estimation (average root-mean-square errors [RMSE] <2.00°) at various flexion-extension speeds. Finally, a hand-tracking system is designed based on a SPAW array. An example application is developed to demonstrate the performance of SPAWs in real-time hand motion tracking in a three-dimensional (3D) virtual environment. To our knowledge, the system detailed in this article is the first to use soft acoustic waveguides to capture human motion. This work is part of an ongoing effort to develop soft sensors using both time and frequency domains, with the goal of extracting decoupled signals from simple sensing structures. As such, it represents a novel and promising path toward soft, simple, and wearable multimodal sensors.

Simulative Evaluation of a Joint-Cartesian Hybrid Motion Mapping for Robot Hands Based on Spatial In-Hand Information

Two sub-problems are typically identified for the replication of human finger motions on artificial hands: the measurement of the motions on the human side and the mapping method of human hand movements (primary hand) on the robotic hand (target hand). In this study, we focus on the second sub-problem. During human to robot hand mapping, ensuring natural motions and predictability for the operator is a difficult task, since it requires the preservation of the Cartesian position of the fingertips and the finger shapes given by the joint values. Several approaches have been presented to deal with this problem, which is still unresolved in general. In this work, we exploit the spatial information available in-hand, in particular, related to the thumb-finger relative position, for combining joint and Cartesian mappings. In this way, it is possible to perform a large range of both volar grasps (where the preservation of finger shapes is more important) and precision grips (where the preservation of fingertip positions is more important) during primary-to-target hand mappings, even if kinematic dissimilarities are present. We therefore report on two specific realizations of this approach: a distance-based hybrid mapping, in which the transition between joint and Cartesian mapping is driven by the approaching of the fingers to the current thumb fingertip position, and a workspace-based hybrid mapping, in which the joint–Cartesian transition is defined on the areas of the workspace in which thumb and fingertips can get in contact. The general mapping approach is presented, and the two realizations are tested. In order to report the results of an evaluation of the proposed mappings for multiple robotic hand kinematic structures (both industrial grippers and anthropomorphic hands, with a variable number of fingers), a simulative evaluation was performed.

Kinematic synergies of hand grasps: a comprehensive study on a large publicly available dataset

Background

Hand grasp patterns require complex coordination. The reduction of the kinematic dimensionality is a key process to study the patterns underlying hand usage and grasping. It allows to define metrics for motor assessment and rehabilitation, to develop assistive devices and prosthesis control methods. Several studies were presented in this field but most of them targeted a limited number of subjects, they focused on postures rather than entire grasping movements and they did not perform separate analysis for the tasks and subjects, which can limit the impact on rehabilitation and assistive applications. This paper provides a comprehensive mapping of synergies from hand grasps targeting activities of daily living. It clarifies several current limits of the field and fosters the development of applications in rehabilitation and assistive robotics.

Methods

In this work, hand kinematic data of 77 subjects, performing up to 20 hand grasps, were acquired with a data glove (a 22-sensor CyberGlove II data glove) and analyzed. Principal Component Analysis (PCA) and hierarchical cluster analysis were used to extract and group kinematic synergies that summarize the coordination patterns available for hand grasps.

Results

Twelve synergies were found to account for > 80% of the overall variation. The first three synergies accounted for more than 50% of the total amount of variance and consisted of: the flexion and adduction of the Metacarpophalangeal joint (MCP) of fingers 3 to 5 (synergy #1), palmar arching and flexion of the wrist (synergy #2) and opposition of the thumb (synergy #3). Further synergies refine movements and have higher variability among subjects.

Conclusion

Kinematic synergies are extracted from a large number of subjects (77) and grasps related to activities of daily living (20). The number of motor modules required to perform the motor tasks is higher than what previously described. Twelve synergies are responsible for most of the variation in hand grasping. The first three are used as primary synergies, while the remaining ones target finer movements (e.g. independence of thumb and index finger). The results generalize the description of hand kinematics, better clarifying several limits of the field and fostering the development of applications in rehabilitation and assistive robotics.

Posture similarity index: a method to compare hand postures in synergy space

Background

The human hand can perform a range of manipulation tasks, from holding a pen to holding a hammer. The central nervous system (CNS) uses different strategies in different manipulation tasks based on task requirements. Attempts to compare postures of the hand have been made for use in robotics and animation industries. In this study, we developed an index called the posture similarity index to quantify the similarity between two human hand postures.

Methods

Twelve right-handed volunteers performed 70 postures, and lifted and held 30 objects (total of 100 different postures, each performed five times). A 16-sensor electromagnetic tracking system captured the kinematics of individual finger phalanges (segments). We modeled the hand as a 21-DoF system and computed the corresponding joint angles. We used principal component analysis to extract kinematic synergies from this 21-DoF data. We developed a posture similarity index (PSI), that represents the similarity between posture in the synergy (Principal component) space. First, we tested the performance of this index using a synthetic dataset. After confirming that it performs well with the synthetic dataset, we used it to analyze the experimental data. Further, we used PSI to identify postures that are “representative” in the sense that they have a greater overlap (in synergy space) with a large number of postures.

Results

Our results confirmed that PSI is a relatively accurate index of similarity in synergy space both with synthetic data and real experimental data. Also, more special postures than common postures were found among “representative” postures.

Conclusion

We developed an index for comparing posture similarity in synergy space and demonstrated its utility by using synthetic dataset and experimental dataset. Besides, we found that “special” postures are actually “special” in the sense that there are more of them in the “representative” postures as identified by our posture similarity index.

Sub-millimetre accurate human hand kinematics: from surface to skeleton

A highly accurate human hand kinematics model and identification are proposed. The model includes the five digits and the palm arc based on mapping function between surface landmarks and estimated joint centres of rotation. Model identification was experimentally performed using a motion tracking system. The evaluation of the marker position estimation error, which is on sub-millimetre level across all digits, underlines model quality and accuracy. Noticeably, with the development of this model, we were able to improve various modelling assumptions from literature and found a basic linear relationship between surface and skeleton rotational angles.

On redundancy resolution of the human thumb, index and middle fingers in cooperative object translation

Redundancy in motion, and synergy in neuromuscular coordination provides significant versatility to the human fingers while performing coordinated grasping and manipulation tasks in several ways. This paper explores how humans may resolve the redundancy in their thumb, index and middle fingers when these digits flex to cooperatively translate a small object toward the palm. It is observed that humans actively employ a secondary subtask of maximizing instantaneous manipulability that helps determine all intermediate finger configurations when performing the primary subtask of following a tip trajectory. This behavior is accurately captured by an inverse kinematic model based on aredundancyparameter. The joint angles get determined unambiguously though the redundancy parameter is shown to depend on the instantaneous finger configurations and also, to attain negative values. Further, this parameter is noted to vary significantly across subjects performing the same kinematic task. The findings, that are based on the experimental finger motion data garnered from 12 subjects, are reckoned to be of significant importance, especially in reference to the challenges in design and control of finger exoskeletons for cooperative manipulation.

Using postural synergies to animate a low-dimensional hand avatar in haptic simulation

A technique to animate a realistic hand avatar with 20 DoFs based on the biomechanics of the human hand is presented. The animation does not use any sensor glove or advanced tracker with markers. The proposed approach is based on the knowledge of a set of kinematic constraints on the model of the hand, referred to as postural synergies, which allows to represent the hand posture using a number of variables lower than the number of joints of the hand model. This low-dimensional set of parameters is estimated from direct measurement of the motion of thumb and index finger tracked using two haptic devices. A kinematic inversion algorithm has been developed, which takes synergies into account and estimates the kinematic configuration of the whole hand, i.e., also of the fingers whose end tips are not directly tracked by the two haptic devices. The hand skin is deformable and its deformation is computed using a linear vertex blending technique. The proposed synergy-based animation of the hand avatar involves only algebraic computations and is suitable for real-time implementation as required in haptics.