Multidigit force control during unconstrained grasping in response to object perturbations

From Human Hand to Grasp Surface Detection, Tracking & Analysis

Hands are paramount for dexterous interactions that humans exhibit in daily life. Understanding the intricacies of human hand-object interactions is therefore necessary. Unfortunately, the limitations of state-of-the-art technologies make capturing the full hand-object complexity unfeasible, giving rise to the need for new technological means to achieve this aim. In this work, we propose an end-to-end framework in which individualized hand models are derived and used to capture quantitative personalized hand-object interaction information, precisely, hand shape, kinematics, and contact surfaces. The results of this study serve as a proof of concept that such a framework can significantly deepen personalized hand-object interaction analyses, providing, in perspective, insights for medical diagnoses and rehabilitation, among others.Clinical relevance- Our work showcases the need to incorporate bespoke human hand models in individualized hand function assessment technologies, as hand-object interaction information is subject-dependent.

A Musculoskeletal Model of the Hand and Wrist Capable of Simulating Functional Tasks

OBJECTIVE

The purpose of this work was to develop an open-source musculoskeletal model of the hand and wrist and to evaluate its performance during simulations of functional tasks.

METHODS

The current model was developed by adapting and expanding upon existing models. An optimal control theory framework that combines forward-dynamics simulations with a simulated-annealing optimization was used to simulate maximum grip and pinch force. Active and passive hand opening were simulated to evaluate coordinated kinematic hand movements.

RESULTS

The model's maximum grip force production matched experimental measures of grip force, force distribution amongst the digits, and displayed sensitivity to wrist flexion. Simulated lateral pinch strength replicated in vivo palmar pinch strength data. Additionally, predicted activations for 7 of 8 muscles fell within variability of EMG data during palmar pinch. The active and passive hand opening simulations predicted reasonable activations and demonstrated passive motion mimicking tenodesis, respectively.

CONCLUSION

This work advances simulation capabilities of hand and wrist models and provides a foundation for future work to build upon.

SIGNIFICANCE

This is the first open-source musculoskeletal model of the hand and wrist to be implemented during both functional kinetic and kinematic tasks. We provide a novel simulation framework to predict maximal grip and pinch force which can be used to evaluate how potential surgical and rehabilitation interventions influence these functional outcomes while requiring minimal experimental data.

Distinct adaptation processes underlie multidigit force coordination for dexterous manipulation

The human sensorimotor system can adapt to various changes in the environmental dynamics by updating motor commands to improve performance after repeated exposure to the same task. However, the characteristics and mechanisms of the adaptation process remain unknown for dexterous manipulation, a unique motor task in which the body physically interacts with the environment with multiple effectors, i.e., digits, in parallel. We addressed this gap by using robotic manipulanda to investigate the changes in the digit force coordination following mechanical perturbation of an object held by tripod grasps. As the participants gradually adapted to lifting the object under perturbations, we quantified two components of digit force coordination. One is the direction-specific manipulation moment that directly counteracts the perturbation, whereas the other one is the direction-independent internal moment that supports the stability and stiffness of the grasp. We found that trial-to-trial improvement of task performance was associated with increased manipulation moment and a gradual decrease of the internal moment. These two moments were characterized by different rates of adaptation. We also examined how these two force coordination components respond to changes in perturbation directions. Importantly, we found that the manipulation moment was sensitive to the extent of repetitive exposure to the previous context that has an opposite perturbation direction, whereas the internal moment did not. However, the internal moment was sensitive to whether the postchange perturbation direction was previously experienced. Our results reveal, for the first time, that two distinct processes underlie the adaptation of multidigit force coordination for dexterous manipulation.NEW & NOTEWORTHY Changes in digit force coordination in multidigit object manipulation were quantified with a novel experimental design in which human participants adapted to mechanical perturbations applied to the object. Our results show that the adaptation of digit force coordination can be characterized by two distinct components that operate at different timescales. We further show that these two components respond to changes in perturbation direction differently.

Spatiotemporal Modeling of Grip Forces Captures Proficiency in Manual Robot Control

New technologies for monitoring grip forces during hand and finger movements in non-standard task contexts have provided unprecedented functional insights into somatosensory cognition. Somatosensory cognition is the basis of our ability to manipulate and transform objects of the physical world and to grasp them with the right amount of force. In previous work, the wireless tracking of grip-force signals recorded from biosensors in the palm of the human hand has permitted us to unravel some of the functional synergies that underlie perceptual and motor learning under conditions of non-standard and essentially unreliable sensory input. This paper builds on this previous work and discusses further, functionally motivated, analyses of individual grip-force data in manual robot control. Grip forces were recorded from various loci in the dominant and non-dominant hands of individuals with wearable wireless sensor technology. Statistical analyses bring to the fore skill-specific temporal variations in thousands of grip forces of a complete novice and a highly proficient expert in manual robot control. A brain-inspired neural network model that uses the output metric of a self-organizing pap with unsupervised winner-take-all learning was run on the sensor output from both hands of each user. The neural network metric expresses the difference between an input representation and its model representation at any given moment in time and reliably captures the differences between novice and expert performance in terms of grip-force variability.Functionally motivated spatiotemporal analysis of individual average grip forces, computed for time windows of constant size in the output of a restricted amount of task-relevant sensors in the dominant (preferred) hand, reveal finger-specific synergies reflecting robotic task skill. The analyses lead the way towards grip-force monitoring in real time. This will permit tracking task skill evolution in trainees, or identify individual proficiency levels in human robot-interaction, which represents unprecedented challenges for perceptual and motor adaptation in environmental contexts of high sensory uncertainty. Cross-disciplinary insights from systems neuroscience and cognitive behavioral science, and the predictive modeling of operator skills using parsimonious Artificial Intelligence (AI), will contribute towards improving the outcome of new types of surgery, in particular the single-port approaches such as NOTES (Natural Orifice Transluminal Endoscopic Surgery) and SILS (Single-Incision Laparoscopic Surgery).

Grip force as a functional window to somatosensory cognition

Analysis of grip force signals tailored to hand and finger movement evolution and changes in grip force control during task execution provide unprecedented functional insight into somatosensory cognition. Somatosensory cognition is the basis of our ability to act upon and to transform the physical world around us, to recognize objects on the basis of touch alone, and to grasp them with the right amount of force for lifting and manipulating them. Recent technology has permitted the wireless monitoring of grip force signals recorded from biosensors in the palm of the human hand to track and trace human grip forces deployed in cognitive tasks executed under conditions of variable sensory (visual, auditory) input. Non-invasive multi-finger grip force sensor technology can be exploited to explore functional interactions between somatosensory brain mechanisms and motor control, in particular during learning a cognitive task where the planning and strategic execution of hand movements is essential. Sensorial and cognitive processes underlying manual skills and/or hand-specific (dominant versus non-dominant hand) behaviors can be studied in a variety of contexts by probing selected measurement loci in the fingers and palm of the human hand. Thousands of sensor data recorded from multiple spatial locations can be approached statistically to breathe functional sense into the forces measured under specific task constraints. Grip force patterns in individual performance profiling may reveal the evolution of grip force control as a direct result of cognitive changes during task learning. Grip forces can be functionally mapped to from-global-to-local coding principles in brain networks governing somatosensory processes for motor control in cognitive tasks leading to a specific task expertise or skill. Under the light of a comprehensive overview of recent discoveries into the functional significance of human grip force variations, perspectives for future studies in cognition, in particular the cognitive control of strategic and task relevant hand movements in complex real-world precision task, are pointed out.

Human Grasp Mechanism Understanding, Human-Inspired Grasp Control and Robotic Grasping Planning for Agricultural Robots

As the end execution tool of agricultural robots, the manipulator directly determines whether the grasping task can be successfully completed. The human hand can adapt to various objects and achieve stable grasping, which is the highest goal for manipulator design and development. Thus, this study combines a multi-sensor fusion tactile glove to simulate manual grasping, explores the mechanism and characteristics of the human hand, and formulates rational grasping plans. According to the shape and size of fruits and vegetables, the grasping gesture library is summarized to facilitate the matching of optimal grasping gestures. By analyzing inter-finger curvature correlations and inter-joint pressure correlations, we investigated the synergistic motion characteristics of the human hand. In addition, the force data were processed by the wavelet transform algorithms and then the thresholds for sliding detection were set to ensure robust grasping. The acceleration law under the interaction with the external environment during grasping was also discussed, including stable movement, accidental collision, and placement of the target position. Finally, according to the analysis and summary of the manual gripping mechanism, the corresponding pre-gripping planning was designed to provide theoretical guidance and ideas for the gripping of robots.

The interaction between motion and texture in the sense of touch

Besides providing information on elementary properties of objects, like texture, roughness, and softness, the sense of touch is also important in building a representation of object movement and the movement of our hands. Neural and behavioral studies shed light on the mechanisms and limits of our sense of touch in the perception of texture and motion, and of its role in the control of movement of our hands. The interplay between the geometrical and mechanical properties of the touched objects, such as shape and texture, the movement of the hand exploring the object, and the motion felt by touch, will be discussed in this article. Interestingly, the interaction between motion and textures can generate perceptual illusions in touch. For example, the orientation and the spacing of the texture elements on a static surface induces the illusion of surface motion when we move our hand on it or can elicit the perception of a curved trajectory during sliding, straight hand movements. In this work we present a multiperspective view that encompasses both the perceptual and the motor aspects, as well as the response of peripheral and central nerve structures, to analyze and better understand the complex mechanisms underpinning the tactile representation of texture and motion. Such a better understanding of the spatiotemporal features of the tactile stimulus can reveal novel transdisciplinary applications in neuroscience and haptics.

A Novel Grip Force Measurement Concept for Tactile Stimulation Mechanisms - Design, Validation, and User Study

In this article, we developed a new grip force measurement concept that allows for embedding tactile stimulation mechanisms in a gripper. This concept is based on a single force sensor to measure the force applied on each side of the gripper, and substantially reduces tactor motion artifacts on force measurement. To test the feasibility of this new concept, we built a device that measures control of grip force in response to a tactile stimulation from a moving tactor. We calibrated and validated our device with a testing setup with a second force sensor over a range of 0 to 20 N without movement of the tactors. We tested the effect of tactor movement on the measured grip force, and measured artifacts of 1% of the measured force. We demonstrated that during the application of dynamically changing grip forces, the average errors were 2.9% and 3.7% for the left and right sides of the gripper, respectively. We characterized the bandwidth, backlash, and noise of our tactile stimulation mechanism. Finally, we conducted a user study and found that in response to tactor movement, participants increased their grip force, the increase was larger for a smaller target force, and depended on the amount of tactile stimulation.

Intention to be force efficient improves high-level anticipatory coordination of finger positions and forces in young and elderly adults

Successful object manipulation requires anticipatory high-level control of finger positions and forces to prevent object slip and tilt. Unlike young adults, who efficiently scale grip forces (GFs) according to surface conditions, old adults were reported to exert excessive grip forces. In this study, we theoretically show how grip force economy depends on the modulation of the centers of pressure on opposing grip surfaces (ΔCoP) according to object properties. In a grasp-to-lift study with young and elderly participants, we investigated how the instruction to lift the object with efficient GF influences the anticipation of torques, ΔCoP and GF control during complex variations of mass distributions and surface properties. Provision of the explicit instruction to strive for force efficiency prompted both age groups to optimize their ΔCoP modulation, although to a lesser degree in the elderly, and also led to a refinement of torque anticipation for a right-sided weight distribution in the young, but not the elderly participants. Consequently, marked drops in GF levels resulted. Furthermore, participants enhanced ΔCoP modulation and lowered GF safety ratios in challenging surface conditions. Higher GF in the elderly was due to decreased skin-surface friction but also worse ΔCoP modulation for lateralized mass distributions when trying to be force efficient. In contrast, safety margins were not elevated in the elderly, suggesting preserved GF control. Our findings demonstrate how task goals influence high-level motor control of object manipulation differentially in young and elderly participants and highlight the necessity to control for both instructions and friction when investigating GF control.NEW & NOTEWORTHY Previous studies have shown that forces are covaried as a function of centers of pressure (CoPs) to exert adequate torques. Here, we demonstrate that force-efficient object manipulation requires the modulation of CoPs and show that providing the instruction to be force efficient and challenging surface conditions elicits a GF safety ratio reduction as well as an optimization of anticipatory CoP modulation and torques in the young and, to a lesser degree, in the elderly.

Role of Tactile Noise in the Control of Digit Normal Force

Whenever we grasp and lift an object, our tactile system provides important information on the contact location and the force exerted on our skin. The human brain integrates signals from multiple sites for a coherent representation of object shape, inertia, weight, and other material properties. It is still an open question whether the control of grasp force occurs at the level of individual fingers or whether it is also influenced by the control and the signals from the other fingers of the same hand. In this work, we approached this question by asking participants to lift, transport, and replace a sensorized object, using three- and four-digit grasp. Tactile input was altered by covering participant's fingertips with a rubber thimble, which reduced the reliability of the tactile sensory input. In different experimental conditions, we covered between one and three fingers opposing the thumb. Normal forces at each finger and the thumb were recorded while grasping and holding the object, with and without the thimble. Consistently with previous studies, reducing tactile sensitivity increased the overall grasping force. The gasping force increased in the covered finger, whereas it did not change from baseline in the remaining bare fingers (except the thumb for equilibrium constraints). Digit placement and object tilt were not systematically affected by rubber thimble conditions. Our results suggest that, in each finger opposing thumb, digit normal force is controlled locally in response to the applied tactile perturbation.

Nanomesh pressure sensor for monitoring finger manipulation without sensory interference

Monitoring of finger manipulation without disturbing the inherent functionalities is critical to understand the sense of natural touch. However, worn or attached sensors affect the natural feeling of the skin. We developed nanomesh pressure sensors that can monitor finger pressure without detectable effects on human sensation. The effect of the sensor on human sensation was quantitatively investigated, and the sensor-applied finger exhibits comparable grip forces with those of the bare finger, even though the attachment of a 2-micrometer-thick polymeric film results in a 14% increase in the grip force after adjusting for friction. Simultaneously, the sensor exhibits an extreme mechanical durability against cyclic shearing and friction greater than hundreds of kilopascals.

Evaluation of Pressure Capacitive Sensors for Application in Grasping and Manipulation Analysis

The analysis of the human grasping and manipulation capabilities is paramount for investigating human sensory-motor control and developing prosthetic and robotic hands resembling the human ones. A viable solution to perform this analysis is to develop instrumented objects measuring the interaction forces with the hand. In this context, the performance of the sensors embedded in the objects is crucial. This paper focuses on the experimental characterization of a class of capacitive pressure sensors suitable for biomechanical analysis. The analysis was performed in three loading conditions (Distributed load, 9 Tips load, and Wave-shaped load, thanks to three different inter-elements) via a traction/compression testing machine. Sensor assessment was also carried out under human- like grasping condition by placing a silicon material with the same properties of prosthetic cosmetic gloves in between the sensor and the inter-element in order to simulate the human skin. Data show that the input-output relationship of the analyzed, sensor is strongly influenced by both the loading condition (i.e., type of inter-element) and the grasping condition (with or without the silicon material). This needs to be taken into account to avoid significant measurement error. To go over this hurdle, the sensors have to be calibrated under each specific condition in order to apply suitable corrections to the sensor output and significantly improve the measurement accuracy.

Taxonomy based analysis of force exchanges during object grasping and manipulation

The flexibility of the human hand in object manipulation is essential for daily life activities, but remains relatively little explored with quantitative methods. On the one hand, recent taxonomies describe qualitatively the classes of hand postures for object grasping and manipulation. On the other hand, the quantitative analysis of hand function has been generally restricted to precision grip (with thumb and index opposition) during lifting tasks. The aim of the present study is to fill the gap between these two kinds of descriptions, by investigating quantitatively the forces exerted by the hand on an instrumented object in a set of representative manipulation tasks. The object was a parallelepiped object able to measure the force exerted on the six faces and its acceleration. The grasping force was estimated from the lateral force and the unloading force from the bottom force. The protocol included eleven tasks with complementary constraints inspired by recent taxonomies: four tasks corresponding to lifting and holding the object with different grasp configurations, and seven to manipulating the object (rotation around each of its axis and translation). The grasping and unloading forces and object rotations were measured during the five phases of the actions: unloading, lifting, holding or manipulation, preparation to deposit, and deposit. The results confirm the tight regulation between grasping and unloading forces during lifting, and extend this to the deposit phase. In addition, they provide a precise description of the regulation of force exchanges during various manipulation tasks spanning representative actions of daily life. The timing of manipulation showed both sequential and overlapping organization of the different sub-actions, and micro-errors could be detected. This phenomenological study confirms the feasibility of using an instrumented object to investigate complex manipulative behavior in humans. This protocol will be used in the future to investigate upper-limb dexterity in patients with sensory-motor impairments.