Advance Planning in Sequential Pick–and–Place Tasks

A review of grasping as the movements of digits in space

It is tempting to describe human reach-to-grasp movements in terms of two, more or less independent visuomotor channels, one relating hand transport to the object’s location and the other relating grip aperture to the object’s size. Our review of experimental work questions this framework for reasons that go beyond noting the dependence between the two channels. Both the lack of effect of size illusions on grip aperture and the finding that the variability in grip aperture does not depend on the object’s size indicate that size information is not used to control grip aperture. An alternative is to describe grip formation as emerging from controlling the movements of the digits in space. Each digit’s trajectory when grasping an object is remarkably similar to its trajectory when moving to tap the same position on its own. The similarity is also evident in the fast responses when the object is displaced. This review develops a new description of the speed-accuracy trade-off for multiple effectors that is applied to grasping. The most direct support for the digit-in-space framework is that prism-induced adaptation of each digit’s tapping movements transfers to that digit’s movements when grasping, leading to changes in grip aperture for adaptation in opposite directions for the two digits. We conclude that although grip aperture and hand transport are convenient variables to describe grasping, treating grasping as movements of the digits in space is a more suitable basis for understanding the neural control of grasping.

Two-step actions in infancy-the TWAIN model

In this paper, we propose a novel model—the TWAIN model—to describe the durations of two-step actions in a reach-to-place task in human infants. Previous research demonstrates that infants and adults plan their actions across multiple steps. They adjust, for instance, the velocity of a reaching action depending on what they intend to do with the object once it is grasped. Despite these findings and irrespective of the larger context in which the action occurs, current models (e.g., Fitts’ law) target single, isolated actions, as, for example, pointing to a goal. In the current paper, we develop and empirically test a more ecologically valid model of two-step action planning. More specifically, 61 18-month olds took part in a reach-to-place task and their reaching and placing durations were measured with a motion-capture system. Our model explained the highest amount of variance in placing duration and outperformed six previously suggested models, when using model comparison. We show that including parameters of the first action step, here the duration of the reaching action, can improve the description of the second action step, here the duration of the placing action. This move towards more ecologically valid models of action planning contributes knowledge as well as a framework for assessing human machine interactions. The TWAIN model provides an updated way to quantify motor learning by the time these abilities develop, which might help to assess performance in typically developing human children.

Motion prediction at low contrast

Accurate motion prediction is fundamental for survival. How does this reconcile with the well-known speed underestimation of low-contrast stimuli? Here we asked whether this contrast-dependent perceptual bias is retained in motion prediction under two different saccadic planning conditions: making a saccade to an occluded moving target, and real-time gaze interaction with multiple moving targets. In a first experiment, observers made a saccade to the mentally extrapolated position of a moving target (imagery condition). In a second experiment, observers had to prevent collisions among multiple moving targets by glancing at them through a gaze-contingent display or by hitting them with the touchpad cursor (interaction condition). In both experiments, target contrast was manipulated. We found that, whereas saccades to the imagined moving target were systematically biased by contrast, the gaze interaction performance, as measured by missed collisions, was generally unaffected - even though low-contrast targets looked slower. Interceptive actions increased at low contrast, but only when the gaze was used for interaction. Thus, perceptual speed underestimation transfers to saccades made to imagined low-contrast targets, without however necessarily being detrimental to effective performance when real-time interaction with multiple targets is required. This differential effect of stimulus contrast suggests that in complex dynamic conditions saccades are rather tolerant to visual speed biases.

Shaping of Reach-to-Grasp Kinematics by Intentions: A Meta-Analysis

When we reach to grasp something, we need to take into account both the properties of the object we are grasping and the intention we have in mind. Previous research has found these constraints to be visible in the reach-to-grasp kinematics, but there is no consensus on which kinematic parameters are the most sensitive. To examine this, a systematic literature search and meta-analyses were performed. The search identified studies assessing how changes in either an object property or a prior intention affect reach-to-grasp kinematics in healthy participants. Hereafter, meta-analyses were conducted using a restricted maximum likelihood random effect model. The meta-analyses showed that changes in both object properties and prior intentions affected reach-to-grasp kinematics. Based on these results, the authors argue for a tripartition of the reach-to-grasp movement in which the accelerating part of the reach is primarily associated with transporting the hand to the object (i.e., extrinsic object properties), the decelerating part of the reach is used as a preparation for object manipulation (i.e., prepare the grasp or the subsequent action), and the grasp is associated with manipulating the object's intrinsic properties, especially object size.

Habit outweighs planning in grasp selection for object manipulation

Object-directed grasping movements are adapted to intended interactions with an object. We address whether adjusting the grasp for object manipulation is controlled habitually, based on past experiences, or by goal-directed planning, based on an evaluation of the expected action outcomes. Therefore, we asked participants to grasp and rotate a dial. In such tasks, participants typically grasp the dial with an excursed, uncomfortable arm posture, which then allows to complete the dial rotation in a comfortable end-state. We extended this task by manipulating the contingency between the orientation of the grasp and the resulting end-state of the arm. A one-step (control) group rotated the dial to a single target. A two-step group rotated the dial to an initial target and then in the opposite direction. A three-step group rotated the dial to the initial target, then in the opposite direction, and then back to the initial target. During practice, the two-step and three-step groups reduced the excursion of their grasps, thus avoiding overly excursed arm postures after the second rotation. When the two-step and three-step groups were asked to execute one-step rotations, their grasps resembled those that were acquired during the two-step and three-step rotations, respectively. However, the carry-over was not complete. This suggests that adjusting grasps for forthcoming object manipulations is controlled by a mixture of habitual and goal-directed processes. In the present experiment, the former contributed approximately twice as much to grasp selection than the latter.

Increased cognitive demands boost the spatial interference effect in bimanual pointing

It is beyond controversy that in bimanual coordination tasks, parameter planning related to the movements of one hand influences the planning and execution of movements simultaneously performed with the other hand. A well-researched example of such bimanual interference is the finding that reaction times tend to be longer when preparing bimanual pointing movements with different amplitudes than for equal amplitude movements. Interestingly, these reaction time costs were found to increase when movement targets were cued symbolically (e.g., using letters) as compared to spatially. Therefore, it was suggested that interference may be primarily related to cue translation and response selection processes rather than resulting from cross-talk at the motor programming level. Here, we argue that spatial interference effects do not necessarily depend on the type of cues used but instead depend on the general task demands (difficulty). In two experiments we show that bimanual interference effects can (1) be abolished in symbolic cueing conditions when highly compatible cues placing minimal demands on response selection processes are used and (2) occur in direct/spatial cueing conditions when a secondary cognitively demanding, but movement-unrelated task is performed. Thus, our findings suggest that whether or not interference effects emerge during movement planning depends on the overall task difficulty and hence the resources available during movement preparation.

Sequential actions: effects of upcoming perceptual and motor tasks on current actions

Reports of spatial interactions between current and upcoming elements in a movement sequence could be due to intentional planning of a "global" action sequence (i.e., strategic effects), or to unintentional motor planning arising from merely paying attention to upcoming target objects (i.e., interference effects). The purpose of this experiment was to determine whether paying attention to an upcoming target object could induce kinematic changes in a current grasping action when there is no strategic advantage associated with such changes. Specifically, participants grasped a rectangular target object in the presence of a second rectangular object that was either ignored, or was the target of a subsequent grasping or perceptual judgment task. Peak grip aperture during the primary grasping action was assessed in relation to the size of the second rectangle. The results revealed an effect of the second rectangle's size only when it was the target of a subsequent perceptual judgment task. This result calls into question the assumption that interactions between current and subsequent elements of an action sequence are necessarily due to strategic movement planning processes and might instead arise from interference arising from merely paying attention to nontarget objects.

The influence of action possibility and end-state comfort on motor imagery of manual action sequences

It has been proposed that the preparation of goal-direct actions involves internal movement simulation, or motor imagery. Evidence suggests that motor imagery is critically involved in the prediction of action consequences and contributes heavily to movement planning processes. The present study examined whether the sensitivity towards end-state comfort and the possibility/impossibility to perform an action sequence are considered during motor imagery. Participants performed a mental rotation task in which two images were simultaneously presented. The image on the left depicted the start posture of a right hand when grasping a bar, while the right image depicted the hand posture at the end of the action sequence. The right image displayed the bar in a vertical orientation with the hand in a comfortable (thumb-up) or in an uncomfortable (thumb-down) posture, while the bar in the left image was rotated in picture plane in steps of 45°. Crucially, the two images formed either a physically possible or physically impossible to perform action sequence. Results revealed strikingly different response time patterns for the two action sequence conditions. In general, response times increased almost monotonically with increasing angular disparity for the possible to perform action sequences. However, slight deviations from this monotonicity were apparent when the sequences contained an uncomfortable as opposed to a comfortable final posture. In contrast, for the impossible sequences, response times did not follow a typical mental rotation function, but instead were uniformly very slow. These findings suggest that both biomechanical constraints (i.e., end-state comfort) and the awareness of the possibility/impossibility to perform an action sequence are considered during motor imagery. We conclude that motor representations contain information about the spatiotemporal movement organization and the possibility of performing an action, which are crucially involved in anticipation and planning of action sequences.

The influence of reducing intermediate target constraints on grasp posture planning during a three-segment object manipulation task

The present experiment examined the influence of final target position on grasp posture planning during a three-segment object manipulation task in which the required object orientation at the first target position was unconstrained. Participants grasped a cylindrical object from a home position, placed it at an intermediate position in a freely chosen orientation, and subsequently placed it at one of four final target positions. Considerable inter-individual differences in initial grasp selection were observed which also led to differences in final grasp postures. Whereas some participants strongly adjusted their initial grasp postures to the final target orientation, and thus showed a preference for end-state comfort, other participants showed virtually no adjustment in initial grasp postures, hence satisfying initial-state comfort. Interestingly, as intermediate grasp postures were similar regardless of initial grasp adjustment, intermediate-state comfort was prioritized by all participants. These results provide further evidence for the interaction of multiple action selection constraints in grasp posture planning during multi-segment object manipulation tasks. Whereas some constraints may take strict precedence in a given task, other constraints may be more flexible and weighted differently among participants. This differentiated weighting leads to task- and subject-specific constraint hierarchies and is reflected in inter-individual differences in grasp selection.

Analysis of human grasping behavior: correlating tasks, objects and grasps

This paper is the second in a two-part series analyzing human grasping behavior during a wide range of unstructured tasks. It investigates the tasks performed during the daily work of two housekeepers and two machinists and correlates grasp type and object properties with the attributes of the tasks being performed. The task or activity is classified according to the force required, the degrees of freedom, and the functional task type. We found that 46 percent of tasks are constrained, where the manipulated object is not allowed to move in a full six degrees of freedom. Analyzing the interrelationships between the grasp, object, and task data show that the best predictors of the grasp type are object size, task constraints, and object mass. Using these attributes, the grasp type can be predicted with 47 percent accuracy. Those parameters likely make useful heuristics for grasp planning systems. The results further suggest the common sub-categorization of grasps into power, intermediate, and precision categories may not be appropriate, indicating that grasps are generally more multi-functional than previously thought. We find large and heavy objects are grasped with a power grasp, but small and lightweight objects are not necessarily grasped with precision grasps-even with grasped object size less than 2 cm and mass less than 20 g, precision grasps are only used 61 percent of the time. These results have important implications for robotic hand design and grasp planners, since it appears while power grasps are frequently used for heavy objects, they can still be quite practical for small, lightweight objects.

Eating interrupted: the effect of intent on hand-to-mouth actions

Evidence from recent neurophysiological studies on nonhuman primates as well as from human behavioral studies suggests that actions with similar kinematic requirements but different end-state goals are supported by separate neural networks. It is unknown whether these different networks supporting seemingly similar reach-to-grasp actions are lateralized, or if they are equally represented in both hemispheres. Recently published behavioral evidence suggests certain networks are lateralized to the left hemisphere. Specifically, when participants used their right hand, their maximum grip aperture (MGA) was smaller when grasping to eat food items than when grasping to place the same items. Left-handed movements showed no difference between tasks. The present study investigates whether the differences between grasp-to-eat and grasp-to-place actions are driven by an intent to eat, or if placing an item into the mouth (sans ingestion) is sufficient to produce asymmetries. Twelve right-handed adults were asked to reach to grasp food items to 1) eat them, 2) place them in a bib, or 3) place them between their lips and then toss them into a nearby receptacle. Participants performed each task with large and small food items, using both their dominant and nondominant hands. The current study replicated the previous finding of smaller MGAs for the eat condition during right-handed but not left-handed grasps. MGAs in the eat and spit conditions did not significantly differ from each other, suggesting that eating and bringing a food item to the mouth both utilize similar motor plans, likely originating within the same neural network. Results are discussed in relation to neurophysiology and development.

Grasp posture planning during multi-segment object manipulation tasks - interaction between cognitive and biomechanical factors

The present study examined adaptations in the planning of initial grasp postures during a multi-segment object manipulation task. Participants performed a grasping and placing task that consisted of one, two, or three movement segments. The position of the targets was manipulated such that the degree of object rotation between the home and temporally proximal positions, and between the temporally proximal and distal target positions, varied. Participants selected initial grasp postures based on the specific requirements of the temporally proximal and temporally distal action segments, and adjustments in initial grasp posture depended on the temporal order of target location. In addition, during the initial stages of the experimental session initial grasp postures were influenced to a larger extent by the demands of the temporally proximal segment. However, over time, participants overcame these cognitive limitations and adjusted their initial grasp postures more strongly to the requirements of the temporally distal segment. Taken together, these results indicate that grasp posture planning is influenced by cognitive and biomechanical factors, and that participants learn to anticipate the task demands of temporally distal task demands, which we hypothesize, reduce the burden on the central nervous system.

Whole-body posture planning in anticipation of a manual prehension task: prospective and retrospective effects

This study examined the extent to which the anticipation of a manual action task influences whole-body postural planning and orientation. Our participants walked up to a drawer, opened the drawer, then grasped and moved an object in the drawer to another location in the same drawer. The starting placement of the object within the drawer and the final placement of the object in the drawer were varied across trials in either a blocked design (i.e., in trials where the same start and end location were repeated consecutively) or in a mixed fashion. Of primary interest was the posture adopted at the moment of grasping the drawer handle before pulling it out prior to the object manipulation task. Of secondary interest was whether there were sequential effects such that postures adopted in preceding trials influenced postures in subsequent trials. The results indicated that the spatial properties of the forthcoming object manipulation influenced both the postures adopted by the participants and the degree to which the drawer was opened, suggesting a prospective effect. In addition, the adopted postures were more consistent in blocked trials than in mixed trials, suggesting an additional retrospective effect. Overall, our findings suggest that motor planning occurs at the level of the whole body, and reflects both prospective and retrospective influences.

Prospective and retrospective effects in a virtual pointing task

Over two decades ago prospective and retrospective effects of posture selection in a sequential task were described for the first time. Since then, both effects have been reproduced in a number of reaching studies. We asked (1) whether retrospective effects would also be found in a sequential pointing task and (2) whether pro/retrospective effects of posture selection would transfer to the end-effector position in the absence of haptic feedback. To this end, we created a sequential, perceptual-motor task in a virtual environment. Participants had to point to a row of targets in the frontal plane in sequential order. In a control experiment, physical targets were placed at the same locations. Results showed that kinematic parameters were similar in the virtual and real environment. Retrospective effects of posture/position were found in neither environment, indicating that pointing movements require lower cognitive planning costs than reaching movements. Prospective effects of posture were found both in the virtual and real environment. Prospective effects of position, on the other hand, were present in the virtual but not in the real environment, indicating that the absence of haptic feedback may result in unconscious shifts of the end-effector position.

Individual differences in motor planning during a multi-segment object manipulation task

Research has demonstrated that people will adopt initially awkward grasps if they afford more comfortable postures at the end of the movement. This end-state comfort effect provides evidence that humans represent future posture states and select appropriate grasps in anticipation of these postures. The purpose of the study was to examine to what extent the final action goal of a task influences motor planning of preceding segments, and whether grasp postures are planned to optimize end-state comfort during a three-segment action sequence in which two objects are manipulated, and participants can select from a continuous range of possible grasp postures. In the current experiment, participants opened a drawer, grasped an object from inside the drawer, and placed it on a table in one of the three target orientations (0°, 90°, or 180° object rotation required). Grasp postures during the initial movement segment (drawer opening) were not influenced by the final action goal (i.e., required target orientation). In contrast, both the intermediate (i.e., object grasping) and the final movement segment (i.e., object placing) were influenced by target orientation. In addition, participants adopted different strategies to achieve the action goal when the object required 180° rotation, with 42 % of participants prioritizing intermediate-state comfort and 58 % prioritizing end-state comfort. The results indicate that individuals optimize task performance by selecting lower level constraints that allow for successful completion of the action goal and that the selection of these constraints is dependent upon contextual, environmental, and internal influences.

Contact points during multidigit grasping of geometric objects

We investigated the choice of contact points during multidigit grasping of different objects. In Experiment 1, cylinders were grasped and lifted. Participants were either instructed as to the number of fingers they should use, ranging from a two-finger grasp to a five-finger grasp, or could grasp with their preferred number of fingers. We found a strong relationship between the position of the fingertips on the object and the number of fingers used. In general, variability in the choice of contact points was low within- as well as between participants. The virtual finger, defined as the geometric mean position of fingers opposing the thumb, was in almost perfect opposition to the thumb, suggesting the formation of a functional unit using all contributing fingers in the grasp. In Experiment 2, four more complex shapes (rectangle, hexagon, pentagon, curved object) were grasped. Although we found some moderate between-participant variability in the choice of contact points, the within-participant variability was again remarkably low. In both experiments, participants showed a strong preference to use four or five fingers during grasping when left with free choice. Taken together, our findings suggest a preplanning of the grasping movement and that grasping results from a coordinated interplay between the fingers contributing to the grasp that cannot be understood as independent digit movements.

Suppression of proprioceptive feedback control in movement sequences through intermediate targets

Simple movements can be seen as building blocks for complex action sequences, and neural control of an action sequence can be expected to preserve some control features of its constituent blocks. It was previously found that during single-joint elbow movements to a single target, the proprioceptive feedback control is initially suppressed, and we tested this feedback suppression in a two-segment sequence during which subjects momentarily slowed down at an intermediate target at a 30° distance (first segment) and then immediately moved another 30° to the final target (second segment). Either the first or second segment was unexpectedly perturbed; the latency of the earliest response to the perturbation in the muscle surface electromyogram was analyzed. The perturbations were delivered either at the onset of each segment or about 0.1 s later. We found that in both segments, the response latency to the late perturbation was shorter than the latency to the early perturbation, which suggests that the proprioceptive feedback control is suppressed in the beginning of each segment. Next, we determined the latency of the response to unexpected perturbations in 30° movements to a single target. We found that the response latency was not significantly different in the movement to a single target and in each segment in the sequence. This result suggests that the initial suppression of the proprioceptive feedback control in movements to single targets is preserved in movements through intermediate targets and supports the idea of modular organization of neural control of movement sequences.