Separate contributions of kinematic and kinetic errors to trajectory and grip force adaptation when transporting novel hand-held loads

Distinct adaptation processes underlie multidigit force coordination for dexterous manipulation

The human sensorimotor system can adapt to various changes in the environmental dynamics by updating motor commands to improve performance after repeated exposure to the same task. However, the characteristics and mechanisms of the adaptation process remain unknown for dexterous manipulation, a unique motor task in which the body physically interacts with the environment with multiple effectors, i.e., digits, in parallel. We addressed this gap by using robotic manipulanda to investigate the changes in the digit force coordination following mechanical perturbation of an object held by tripod grasps. As the participants gradually adapted to lifting the object under perturbations, we quantified two components of digit force coordination. One is the direction-specific manipulation moment that directly counteracts the perturbation, whereas the other one is the direction-independent internal moment that supports the stability and stiffness of the grasp. We found that trial-to-trial improvement of task performance was associated with increased manipulation moment and a gradual decrease of the internal moment. These two moments were characterized by different rates of adaptation. We also examined how these two force coordination components respond to changes in perturbation directions. Importantly, we found that the manipulation moment was sensitive to the extent of repetitive exposure to the previous context that has an opposite perturbation direction, whereas the internal moment did not. However, the internal moment was sensitive to whether the postchange perturbation direction was previously experienced. Our results reveal, for the first time, that two distinct processes underlie the adaptation of multidigit force coordination for dexterous manipulation.NEW & NOTEWORTHY Changes in digit force coordination in multidigit object manipulation were quantified with a novel experimental design in which human participants adapted to mechanical perturbations applied to the object. Our results show that the adaptation of digit force coordination can be characterized by two distinct components that operate at different timescales. We further show that these two components respond to changes in perturbation direction differently.

Grip Force Control During Virtual Interaction With Deformable and Rigid Objects Via a Haptic Gripper

Most studies of grip force control focus on the manipulation of rigid objects. However, in virtual and teleoperation applications, objects are often elastic in the pinch degree-of-freedom, and are manipulated using a handle that presents haptic feedback to the user. When designing controllers for haptic grippers, it is crucial to understand how grip force is controlled when manipulating rigid and deformable objects. Here, we used a virtual teleoperation setup with a haptic gripper interface to investigate grip force control in virtual environments. Ten participants lifted virtual objects and performed vertical, cyclic motions using a haptic gripper. We manipulated the control signal to the virtual interface, the haptic properties of the gripper, and the visual properties of the virtual objects to test their effect on the grip force control. We found that participants modulated their grip force as a function of the anticipated load force in all of the experimental conditions. The control signal and properties of the haptic gripper, but not the visual properties of the objects, affected the baseline and the extent of the grip force modulation. These results can provide design guidelines for haptic grippers that elicit natural grasping in virtual and bilateral teleoperation applications.

The Effect of Variability in Stiffness on Perception and Grip Force Adjustment

Haptic information can be used to create our perception of the stiffness of objects and to regulate grip force. Introducing noise into sensory inputs can create uncertainty, yet a method of creating haptic uncertainty without distorting the haptic information has yet to be discovered. Toward this end, in this article, we investigated the effect of varying haptic information between consecutive interactions with an elastic force field on stiffness perception and grip force control. In a stiffness discrimination task, participants interacted with force fields multiple times. Low, medium, and high variability levels were created by drawing the stiffness level applied in each consecutive interaction within a trial from normal distributions. Perceptual haptic uncertainty was created only by the medium variability level. Moreover, all the variability levels affected the grip force control: the modulation of the grip force with the load force decreased with repeated interactions with the force field, whereas no change in the baseline grip force was observed. Additionally, we ascertained that participants formed their perceived stiffness by calculating a weighted average of the different stiffness levels applied by a given force field. We conclude that the medium variability level can be effective in inducing uncertainty in both perception and action.

Distinct adaptation patterns between grip dynamics and arm kinematics when the body is upside-down

In humans, practically all movements are learnt and performed in a constant gravitational field. Yet, studies on arm movements and object manipulation in parabolic flight have highlighted very fast sensorimotor adaptations to altered gravity environments. Here, we wondered if the motor adjustments observed in those altered gravity environments could also be observed on Earth in a situation where the body is upside-down. To address this question, we asked participants to perform rhythmic arm movements in two different body postures (right-side-up and upside-down) while holding an object in precision grip. Analyses of grip-load force coordination and of movement kinematics revealed distinct adaptation patterns between grip and arm control. Grip force and load force were tightly synchronized from the first movements performed in upside-down posture, reflecting a malleable allocentric grip control. In contrast, velocity profiles showed a more progressive adaptation to the upside-down posture and reflected an egocentric planning of arm kinematics. In addition to suggesting distinct mechanisms between grip dynamics and arm kinematics for adaptation to novel contexts, these results also suggest the existence of general mechanisms underlying gravity-dependent motor adaptation that can be used for fast sensorimotor coordination across different postures on Earth and, incidentally, across different gravitational conditions in parabolic flights, in human centrifuges, or in Space.NEW & NOTEWORTHY During rhythmic arm movements performed in an upside-down posture, grip control adapted very quickly, but kinematics adaptation was more progressive. Our results suggest that grip control and movement kinematics planning might operate in different reference frames. Moreover, by comparing our results with previous results from parabolic flight studies, we propose that a common mechanism underlies adaptation to unfamiliar body postures and adaptation to altered gravity.

The Representation of Finger Movement and Force in Human Motor and Premotor Cortices

The ability to grasp and manipulate objects requires controlling both finger movement kinematics and isometric force in rapid succession. Previous work suggests that these behavioral modes are controlled separately, but it is unknown whether the cerebral cortex represents them differently. Here, we asked the question of how movement and force were represented cortically, when executed sequentially with the same finger. We recorded high-density electrocorticography (ECoG) from the motor and premotor cortices of seven human subjects performing a movement-force motor task. We decoded finger movement [0.7 ± 0.3 fractional variance accounted for (FVAF)] and force (0.7 ± 0.2 FVAF) with high accuracy, yet found different spatial representations. In addition, we used a state-of-the-art deep learning method to uncover smooth, repeatable trajectories through ECoG state space during the movement-force task. We also summarized ECoG across trials and participants by developing a new metric, the neural vector angle (NVA). Thus, state-space techniques can help to investigate broad cortical networks. Finally, we were able to classify the behavioral mode from neural signals with high accuracy (90 ± 6%). Thus, finger movement and force appear to have distinct representations in motor/premotor cortices. These results inform our understanding of the neural control of movement, as well as the design of grasp brain-machine interfaces (BMIs).

The effect of tactile augmentation on manipulation and grip force control during force-field adaptation

Background

When exposed to a novel dynamic perturbation, participants adapt by changing their movements’ dynamics. This adaptation is achieved by constructing an internal representation of the perturbation, which allows for applying forces that compensate for the novel external conditions. To form an internal representation, the sensorimotor system gathers and integrates sensory inputs, including kinesthetic and tactile information about the external load. The relative contribution of the kinesthetic and tactile information in force-field adaptation is poorly understood.

Methods

In this study, we set out to establish the effect of augmented tactile information on adaptation to force-field. Two groups of participants received a velocity-dependent tangential skin deformation from a custom-built skin-stretch device together with a velocity-dependent force-field from a kinesthetic haptic device. One group experienced a skin deformation in the same direction of the force, and the other in the opposite direction. A third group received only the velocity-dependent force-field.

Results

We found that adding a skin deformation did not affect the kinematics of the movement during adaptation. However, participants who received skin deformation in the opposite direction adapted their manipulation forces faster and to a greater extent than those who received skin deformation in the same direction of the force. In addition, we found that skin deformation in the same direction to the force-field caused an increase in the applied grip-force per amount of load force, both in response and in anticipation of the stretch, compared to the other two groups.

Conclusions

Augmented tactile information affects the internal representations for the control of manipulation and grip forces, and these internal representations are likely updated via distinct mechanisms. We discuss the implications of these results for assistive and rehabilitation devices.

Adaptivity of End Effector Motor Control Under Different Sensory Conditions: Experiments With Humans in Virtual Reality and Robotic Applications

The investigation of human perception and movement kinematics during manipulation tasks provides insights that can be applied in the design of robotic systems in order to perform human-like manipulations in different contexts and with different performance requirements. In this paper we investigate control in a motor task, in which a tool is moved vertically until it touches a support surface. We evaluate how acoustic and haptic sensory information generated at the moment of contact modulates the kinematic parameters of the movement. Experimental results show differences in the achieved motor control precision and adaptation rate across conditions. We describe how the experimental results can be used in robotics applications in the fields of unsupervised learning, supervised learning from human demonstrators and teleoperations.

Emergent Synergistic Grasp-Like Behavior in a Visuomotor Joint Action Task: Evidence for Internal Forward Models as Building Blocks of Human Interactions

Central to the mechanistic understanding of the human mind is to clarify how cognitive functions arise from simpler sensory and motor functions. A longstanding assumption is that forward models used by sensorimotor control to anticipate actions also serve to incorporate other people's actions and intentions, and give rise to sensorimotor interactions between people, and even abstract forms of interactions. That is, forward models could aid core aspects of human social cognition. To test whether forward models can be used to coordinate interactions, here we measured the movements of pairs of participants in a novel joint action task. For the task they collaborated to lift an object, each of them using fingers of one hand to push against the object from opposite sides, just like a single person would use two hands to grasp the object bimanually. Perturbations of the object were applied randomly as they are known to impact grasp-specific movement components in common grasping tasks. We found that co-actors quickly learned to make grasp-like movements with grasp components that showed coordination on average based on action observation of peak deviation and velocity of their partner's trajectories. Our data suggest that co-actors adopted pre-existing bimanual grasp programs for their own body to use forward models of their partner's effectors. This is consistent with the long-held assumption that human higher-order cognitive functions may take advantage of sensorimotor forward models to plan social behavior. New and Noteworthy: Taking an approach of sensorimotor neuroscience, our work provides evidence for a long-held belief that the coordination of physical as well as abstract interactions between people originates from certain sensorimotor control processes that form mental representations of people's bodies and actions, called forward models. With a new joint action paradigm and several new analysis approaches we show that, indeed, people coordinate each other's interactions based on forward models and mutual action observation.

Visual delay affects force scaling and weight perception during object lifting in virtual reality

Lifting an object requires precise scaling of fingertip forces based on a prediction of object weight. At object contact, a series of tactile and visual events arise that need to be rapidly processed online to fine-tune the planned motor commands for lifting the object. The brain mechanisms underlying multisensory integration serially at transient sensorimotor events, a general feature of actions requiring hand-object interactions, are not yet understood. In this study we tested the relative weighting between haptic and visual signals when they are integrated online into the motor command. We used a new virtual reality setup to desynchronize visual feedback from haptics, which allowed us to probe the relative contribution of haptics and vision in driving participants’ movements when they grasped virtual objects simulated by two force-feedback robots. We found that visual delay changed the profile of fingertip force generation and led participants to perceive objects as heavier than when lifts were performed without visual delay. We further modeled the effect of vision on motor output by manipulating the extent to which delayed visual events could bias the force profile, which allowed us to determine the specific weighting the brain assigns to haptics and vision. Our results show for the first time how visuo-haptic integration is processed at discrete sensorimotor events for controlling object-lifting dynamics and further highlight the organization of multisensory signals online for controlling action and perception.

NEW & NOTEWORTHY Dexterous hand movements require rapid integration of information from different senses, in particular touch and vision, at different key time points as movement unfolds. The relative weighting between vision and haptics for object manipulation is unknown. We used object lifting in virtual reality to desynchronize visual and haptic feedback and find out their relative weightings. Our findings shed light on how rapid multisensory integration is processed over a series of discrete sensorimotor control points.

State-Based Delay Representation and Its Transfer from a Game of Pong to Reaching and Tracking

To accurately estimate the state of the body, the nervous system needs to account for delays between signals from different sensory modalities. To investigate how such delays may be represented in the sensorimotor system, we asked human participants to play a virtual pong game in which the movement of the virtual paddle was delayed with respect to their hand movement. We tested the representation of this new mapping between the hand and the delayed paddle by examining transfer of adaptation to blind reaching and blind tracking tasks. These blind tasks enabled to capture the representation in feedforward mechanisms of movement control. A Time Representation of the delay is an estimation of the actual time lag between hand and paddle movements. A State Representation is a representation of delay using current state variables: the distance between the paddle and the ball originating from the delay may be considered as a spatial shift; the low sensitivity in the response of the paddle may be interpreted as a minifying gain; and the lag may be attributed to a mechanical resistance that influences paddle's movement. We found that the effects of prolonged exposure to the delayed feedback transferred to blind reaching and tracking tasks and caused participants to exhibit hypermetric movements. These results, together with simulations of our representation models, suggest that delay is not represented based on time, but rather as a spatial gain change in visuomotor mapping.

Credit assignment between body and object probed by an object transportation task

It has been proposed that learning from movement errors involves a credit assignment problem: did I misestimate properties of the object or those of my body? For example, an overestimate of arm strength and an underestimate of the weight of a coffee cup can both lead to coffee spills. Though previous studies have found signs of simultaneous learning of the object and of the body during object manipulation, there is little behavioral evidence about their quantitative relation. Here we employed a novel weight-transportation task, in which participants lift the first cup filled with liquid while assessing their learning from errors. Specifically, we examined their transfer of learning when switching to a contralateral hand, the second identical cup, or switching both hands and cups. By comparing these transfer behaviors, we found that 25% of the learning was attributed to the object (simply because of the use of the same cup) and 58% of the learning was attributed to the body (simply because of the use of the same hand). The nervous system thus seems to partition the learning of object manipulation between the object and the body.

Grip force when reaching with target uncertainty provides evidence for motor optimization over averaging

When presented with competing potential reach targets and required to launch a movement before knowing which one will be cued as the target, people initially reach in the average target direction. Although this spatial averaging could arise from executing a weighted average of motor plans for the potential targets, it could also arise from planning a single, optimal movement. To test between these alternatives we used a task in which participants were required to reach to either a single target or towards two potential targets while grasping an object. A robotic device applied a lateral elastic load to the object requiring large grip forces for reaches to targets either side of midline and a minimal grip force for midline movements. As expected, in trials with two targets located either side of midline, participants initially reached straight ahead. Critically, on these trials the initial grip force was minimal, appropriate for the midline movement, and not the average of the large grip forces required for movements to the individual targets. These results indicate that under conditions of target uncertainty, people do not execute an average of planned actions but rather a single movement that optimizes motor costs.

Eye tracking a self-moved target with complex hand-target dynamics

Previous work has shown that the ability to track with the eye a moving target is substantially improved when the target is self-moved by the subject's hand compared with when being externally moved. Here, we explored a situation in which the mapping between hand movement and target motion was perturbed by simulating an elastic relationship between the hand and target. Our objective was to determine whether the predictive mechanisms driving eye-hand coordination could be updated to accommodate this complex hand-target dynamics. To fully appreciate the behavioral effects of this perturbation, we compared eye tracking performance when self-moving a target with a rigid mapping (simple) and a spring mapping as well as when the subject tracked target trajectories that he/she had previously generated when using the rigid or spring mapping. Concerning the rigid mapping, our results confirmed that smooth pursuit was more accurate when the target was self-moved than externally moved. In contrast, with the spring mapping, eye tracking had initially similar low spatial accuracy (though shorter temporal lag) in the self versus externally moved conditions. However, within ∼5 min of practice, smooth pursuit improved in the self-moved spring condition, up to a level similar to the self-moved rigid condition. Subsequently, when the mapping unexpectedly switched from spring to rigid, the eye initially followed the expected target trajectory and not the real one, thereby suggesting that subjects used an internal representation of the new hand-target dynamics. Overall, these results emphasize the stunning adaptability of smooth pursuit when self-maneuvering objects with complex dynamics.

Rapid Visuomotor Corrective Responses during Transport of Hand-Held Objects Incorporate Novel Object Dynamics

Numerous studies have shown that people are adept at learning novel object dynamics, linking applied force and motion, when performing reaching movements with hand-held objects. Here we investigated whether the control of rapid corrective arm responses, elicited in response to visual perturbations, has access to such newly acquired knowledge of object dynamics. Participants first learned to make reaching movements while grasping an object subjected to complex load forces that depended on the distance and angle of the hand from the start position. During a subsequent test phase, we examined grip and load force coordination during corrective arm movements elicited (within ∼150 ms) in response to viewed sudden lateral shifts (1.5 cm) in target or object position. We hypothesized that, if knowledge of object dynamics is incorporated in the control of the corrective responses, grip force changes would anticipate the unusual load force changes associated with the corrective arm movements so as to support grasp stability. Indeed, we found that the participants generated grip force adjustments tightly coupled, both spatially and temporally, to the load force changes associated with the arm movement corrections. We submit that recently learned novel object dynamics are effectively integrated into sensorimotor control policies that support rapid visually driven arm corrective actions during transport of hand held objects. Significance statement: Previous studies have demonstrated that the motor system can learn, and make use of, internal models of object dynamics to generate feedforward motor commands. However, it is not known whether such internal models are incorporated into rapid, automatic arm movement corrections that compensate for errors that arise during movement. Here we demonstrate, for the first time, that internal models of novel object dynamics are integrated into rapid corrective arm movements made in response to visuomotor perturbations that, importantly, do not directly perturb the object.

Inertial torque during reaching directly impacts grip-force adaptation to weightless objects

A hallmark of movement control expressed by healthy humans is the ability to gradually improve motor performance through learning. In the context of object manipulation, previous work has shown that the presence of a torque load has a direct impact on grip-force control, characterized by a significantly slower grip-force adjustment across lifting movements. The origin of this slower adaptation rate remains unclear. On the one hand, information about tangential constraints during stationary holding may be difficult to extract in the presence of a torque. On the other hand, inertial torque experienced during movement may also potentially disrupt the grip-force adjustments, as the dynamical constraints clearly differ from the situation when no torque load is present. To address the influence of inertial torque loads, we instructed healthy adults to perform visually guided reaching movements in weightlessness while holding an unbalanced object relative to the grip axis. Weightlessness offered the possibility to remove gravitational constraints and isolate the effect of movement-related feedback on grip force adjustments. Grip-force adaptation rates were compared with a control group who manipulated a balanced object without any torque load and also in weightlessness. Our results clearly show that grip-force adaptation in the presence of a torque load is significantly slower, which suggests that the presence of torque loads experienced during movement may alter our internal estimates of how much force is required to hold an unbalanced object stable. This observation may explain why grasping objects around the expected location of the center of mass is such an important component of planning and control of manipulation tasks.

Robot-assisted surgery: an emerging platform for human neuroscience research

Classic studies in human sensorimotor control use simplified tasks to uncover fundamental control strategies employed by the nervous system. Such simple tasks are critical for isolating specific features of motor, sensory, or cognitive processes, and for inferring causality between these features and observed behavioral changes. However, it remains unclear how these theories translate to complex sensorimotor tasks or to natural behaviors. Part of the difficulty in performing such experiments has been the lack of appropriate tools for measuring complex motor skills in real-world contexts. Robot-assisted surgery (RAS) provides an opportunity to overcome these challenges by enabling unobtrusive measurements of user behavior. In addition, a continuum of tasks with varying complexity-from simple tasks such as those in classic studies to highly complex tasks such as a surgical procedure-can be studied using RAS platforms. Finally, RAS includes a diverse participant population of inexperienced users all the way to expert surgeons. In this perspective, we illustrate how the characteristics of RAS systems make them compelling platforms to extend many theories in human neuroscience, as well as, to develop new theories altogether.

The effect of force feedback delay on stiffness perception and grip force modulation during tool-mediated interaction with elastic force fields

During interaction with objects, we form an internal representation of their mechanical properties. This representation is used for perception and for guiding actions, such as in precision grip, where grip force is modulated with the predicted load forces. In this study, we explored the relationship between grip force adjustment and perception of stiffness during interaction with linear elastic force fields. In a forced-choice paradigm, participants probed pairs of virtual force fields while grasping a force sensor that was attached to a haptic device. For each pair, they were asked which field had higher level of stiffness. In half of the pairs, the force feedback of one of the fields was delayed. Participants underestimated the stiffness of the delayed field relatively to the nondelayed, but their grip force characteristics were similar in both conditions. We analyzed the magnitude of the grip force and the lag between the grip force and the load force in the exploratory probing movements within each trial. Right before answering which force field had higher level of stiffness, both magnitude and lag were similar between delayed and nondelayed force fields. These results suggest that an accurate internal representation of environment stiffness and time delay was used for adjusting the grip force. However, this representation did not help in eliminating the bias in stiffness perception. We argue that during performance of a perceptual task that is based on proprioceptive feedback, separate neural mechanisms are responsible for perception and action-related computations in the brain.

Extracting kinetic information from human motor cortical signals

Brain machine interfaces (BMIs) have the potential to provide intuitive control of neuroprostheses to restore grasp to patients with paralyzed or amputated upper limbs. For these neuroprostheses to function, the ability to accurately control grasp force is critical. Grasp force can be decoded from neuronal spikes in monkeys, and hand kinematics can be decoded using electrocorticogram (ECoG) signals recorded from the surface of the human motor cortex. We hypothesized that kinetic information about grasping could also be extracted from ECoG, and sought to decode continuously-graded grasp force. In this study, we decoded isometric pinch force with high accuracy from ECoG in 10 human subjects. The predicted signals explained from 22% to 88% (60 ± 6%, mean ± SE) of the variance in the actual force generated. We also decoded muscle activity in the finger flexors, with similar accuracy to force decoding. We found that high gamma band and time domain features of the ECoG signal were most informative about kinetics, similar to our previous findings with intracortical LFPs. In addition, we found that peak cortical representations of force applied by the index and little fingers were separated by only about 4mm. Thus, ECoG can be used to decode not only kinematics, but also kinetics of movement. This is an important step toward restoring intuitively-controlled grasp to impaired patients.

Grip force control during virtual object interaction: effect of force feedback,accuracy demands, and training

When grasping and manipulating objects, people are able to efficiently modulate their grip force according to the experienced load force. Effective grip force control involves providing enough grip force to prevent the object from slipping, while avoiding excessive force to avoid damage and fatigue. During indirect object manipulation via teleoperation systems or in virtual environments, users often receive limited somatosensory feedback about objects with which they interact. This study examines the effects of force feedback, accuracy demands, and training on grip force control during object interaction in a virtual environment. The task required subjects to grasp and move a virtual object while tracking a target. When force feedback was not provided, subjects failed to couple grip and load force, a capability fundamental to direct object interaction. Subjects also exerted larger grip force without force feedback and when accuracy demands of the tracking task were high. In addition, the presence or absence of force feedback during training affected subsequent performance, even when the feedback condition was switched. Subjects' grip force control remained reminiscent of their employed grip during the initial training. These results motivate the use of force feedback during telemanipulation and highlight the effect of force feedback during training.