A neuroeconomics approach to inferring utility functions in sensorimotor control

Physical effort precrastination determines preference in an isometric task

How the brain decides when to invest effort is a central question in neuroscience. When asked to walk a mile to a destination, would you choose a path with a hill at the beginning or the end? The traditional view of effort suggests we should be indifferent-all joules are equal so long as it does not interfere with accomplishing the goal. Yet, when total joules are equal across movement decisions, the brain's sensitivity to the temporal profile of effort investment remains poorly understood. Here, we sought to parse the interaction of time and physical effort by comparing subjective preferences in an isometric arm-pushing task that varied the duration and timing of high and low effort. Subjects were presented with a series of two-alternative forced choices, where they chose the force profile they would rather complete. Subjects preferred earlier physical effort (i.e., to precrastinate) but were idiosyncratic about preference for task timing. A model of subjective utility that includes physical effort costs, task costs, and independent temporal sensitivity factors described subject preferences best. Interestingly, deliberation time and response vigor covary with the same subjective utility model for preference, suggesting a utility that underlies both decision making and motor control. These results suggest physical effort costs are temporally sensitive, with earlier investment of effort preferred to later investment, and that the representation of effort is based on not only the total energy required but also when it is required to invest that energy.NEW & NOTEWORTHY We use a novel paradigm that differentiates physical effort costs, task costs, and time, where subjects choose between isometric arm-pushing tasks. Subjects prefer high physical effort earlier than later. They also decide faster and respond more vigorously the greater their preference. We find a generalizable subjective utility model that includes independent time-sensitivity of physical effort and task costs. Together, we demonstrate that subjective effort includes not only the total effort invested but also its timing.

Reward invigorates isometric gripping actions

Individuals exhibit a propensity to move faster toward more rewarding stimuli. While this phenomenon has been observed in movements, the effect of reward on implicit control of isometric actions, like gripping or grasping, is relatively unknown. How reward-related invigoration generalizes to other effortful actions is an important question. Reward invigorates reaching movements and saccades, supporting the idea that reward pays the additional effort cost of moving faster. Effort in isometric force generation is less understood, so here we ask whether and how reward-related invigoration generalizes to isometric force gripping. And if so, what implicit characteristics of gripping change when there is a prospect of reward? Participants (N=19) gripped a force transducer and the force applied was mapped to radial position of an onscreen cursor. Each trial, a target appeared in one of four locations; increasing grip force moved the cursor toward the target. The gripping action was interchangeable for all target positions. In each block of 100 trials, one target was consistently rewarded, while the other targets were not. When gripping to acquire the rewarded target, participants reacted faster, generated force more rapidly and to a greater extent, while intriguingly maintaining the same accuracy and integral of force over time. These findings support the generalization of reward-related invigoration in isometric force tasks, and that the brain exquisitely trades-off reward and effort costs to obtain reward more rapidly without compromising accuracy or more effort costs than necessary.

NEW & NOTEWORTHY

Gripping actions are important for day-to-day tasks, for medical diagnostics like strength and force control, and for choice selection in decision-making experiments. Comparing isometric gripping responses to reward and nonreward cues, we observed reward-based invigoration mediated by selective increases in effort. These findings can be leveraged to provide additional insight into the decision making process, and better understand the effect of reward on movement vigor and the implicit control of accuracy.

Explicit effort may not influence perceptuomotor decision-making

Many decisions that humans make are enacted by the action system. For example, humans use reach-to-grasp movements when making perceptuomotor decisions between and obtaining fruits of varying quality from a pile. Recent work suggests that the characteristics of each action alternative may influence the decision itself-there may be a bias away from making perceptuomotor alternatives associated with high effort when participants are unaware of the effort differences between responses. The present study examined if perceptuomotor decisions were influenced by explicit reaching effort differences. Neurotypical human participants were presented with random dot motion stimuli in which most dots moved in random directions and varying percentages of remaining dots moved coherently left- or rightward. Participants reported leftward motion judgements by performing leftward (or left hand) reaching movements and rightward motion judgements by performing rightward (or right hand) reaching movements. A resistance band was affixed to participants' wrists and to the table in different configurations. The configurations allowed for one movement/motion direction judgement to always require stretching of the band and, therefore, require relatively more effort. Across a set of experiments, the response context (i.e. selecting directions within a limb or selecting between limbs) and the effort difference between responses were manipulated. Overall, no experiment revealed a bias away from the perceptuomotor decision associated with high effort. Based on these results, it is concluded that, in this biomechanical context, explicit effort may not influence perceptuomotor decision-making and may point to a contextual influence of action effort on perceptuomotor decision-making.

Energy expenditure does not solely explain step length-width choices during walking

Healthy young adults have a most preferred walking speed, step length and step width that are close to energetically optimal. However, people can choose to walk with a multitude of different step lengths and widths, which can vary in both energy expenditure and preference. Here, we further investigated step length-width preferences and their relationship to energy expenditure. In line with a growing body of research, we hypothesized that people's preferred stepping patterns would not be fully explained by metabolic energy expenditure. To test this hypothesis, we used a two-alternative forced-choice paradigm. Fifteen participants walked on an oversized treadmill. Each trial, participants performed two prescribed stepping patterns and then chose the pattern they preferred. Over time, we adapted the choices such that there was 50% chance of choosing one pattern over another (equally preferred). If people's preferences are based solely on metabolic energy expenditure, then these equally preferred stepping patterns should have equal energy expenditure. In contrast, we found that energy expenditure differed across equally preferred step length-width patterns (P<0.001). On average, longer steps with higher energy expenditure were preferred over shorter and wider steps with lower energy expenditure (P<0.001). We also asked participants to rank a set of shorter, wider and longer steps from most preferred to least preferred, and from most energy expended to least energy expended. Only 7/15 participants had the same rankings for their preferences and perceived energy expenditure. Our results suggest that energy expenditure is not the only factor influencing a person's conscious gait choices.

Dissociating the Impact of Movement Time and Energy Costs on Decision-Making and Action Initiation in Humans

Recent theories and data suggest that adapted behavior involves economic computations during which multiple trade-offs between reward value, accuracy requirement, energy expenditure, and elapsing time are solved so as to obtain rewards as soon as possible while spending the least possible amount of energy. However, the relative impact of movement energy and duration costs on perceptual decision-making and movement initiation is poorly understood. Here, we tested 31 healthy subjects on a perceptual decision-making task in which they executed reaching movements to report probabilistic choices. In distinct blocks of trials, the reaching duration (“Time” condition) and energy (“Effort” condition) costs were independently varied compared to a “Reference” block, while decision difficulty was maintained similar at the block level. Participants also performed a simple delayed-reaching (DR) task aimed at estimating movement initiation duration in each motor condition. Results in that DR task show that long duration movements extended reaction times (RTs) in most subjects, whereas energy-consuming movements led to mixed effects on RTs. In the decision task, about half of the subjects decreased their decision durations (DDs) in the Time condition, while the impact of energy on DDs were again mixed across subjects. Decision accuracy was overall similar across motor conditions. These results indicate that movement duration and, to a lesser extent, energy expenditure, idiosyncratically affect perceptual decision-making and action initiation. We propose that subjects who shortened their choices in the time-consuming condition of the decision task did so to limit a drop of reward rate.

Using metabolic energy to quantify the subjective value of physical effort

Economists have known for centuries that to understand an individual's decisions, we must consider not only the objective value of the goal at stake, but its subjective value as well. However, achieving that goal ultimately requires expenditure of effort. Surprisingly, despite the ubiquitous role of effort in decision-making and movement, we currently do not understand how effort is subjectively valued in daily movements. Part of the difficulty arises from the lack of an objective measure of effort. Here, we use a physiological approach to address this knowledge gap. We quantified objective effort costs by measuring metabolic cost via expired gas analysis as participants performed a reaching task against increasing resistance. We then used neuroeconomic methods to quantify each individual's subjective valuation of effort. Rather than the diminishing sensitivity observed in reward valuation, effort was valued objectively, on average. This is significantly less than the near-quadratic sensitivity to effort observed previously in force-based motor tasks. Moreover, there was significant inter-individual variability with many participants undervaluing or overvaluing effort. These findings demonstrate that in contrast with monetary decisions in which subjective value exhibits diminishing marginal returns, effort costs are valued more objectively in low-effort reaching movements common in daily life.

Dopamine Manipulation Affects Response Vigor Independently of Opportunity Cost

Dopamine is known to be involved in regulating effort investment in relation to reward, and the disruption of this mechanism is thought to be central in some pathological situations such as Parkinson's disease, addiction, and depression. According to an influential model, dopamine plays this role by encoding the opportunity cost, i.e., the average value of forfeited actions, which is an important parameter to take into account when making decisions about which action to undertake and how fast to execute it. We tested this hypothesis by asking healthy human participants to perform two effort-based decision-making tasks, following either placebo or levodopa intake in a double blind within-subject protocol. In the effort-constrained task, there was a trade-off between the amount of force exerted and the time spent in executing the task, such that investing more effort decreased the opportunity cost. In the time-constrained task, the effort duration was constant, but exerting more force allowed the subject to earn more substantial reward instead of saving time. Contrary to the model predictions, we found that levodopa caused an increase in the force exerted only in the time-constrained task, in which there was no trade-off between effort and opportunity cost. In addition, a computational model showed that dopamine manipulation left the opportunity cost factor unaffected but altered the ratio between the effort cost and reinforcement value. These findings suggest that dopamine does not represent the opportunity cost but rather modulates how much effort a given reward is worth.

SIGNIFICANCE STATEMENT

Dopamine has been proposed in a prevalent theory to signal the average reward rate, used to estimate the cost of investing time in an action, also referred to as opportunity cost. We contrasted the effect of dopamine manipulation in healthy participants in two tasks, in which increasing response vigor (i.e., the amount of effort invested in an action) allowed either to save time or to earn more reward. We found that levodopa-a synthetic precursor of dopamine-increases response vigor only in the latter situation, demonstrating that, rather than the opportunity cost, dopamine is involved in computing the expected value of effort.

Perceived effort for motor control and decision-making

How effort is internally quantified and how it influences both movement generation and decisions between potential movements are 2 difficult questions to answer. Physical costs are known to influence motor control and decision-making, yet we lack a general, principled characterization of how the perception of effort operates across tasks and conditions. Morel and colleagues introduce an insightful approach to that end, assessing effort indifference points and presenting a quadratic law between perceived effort and force production.

What makes a reach movement effortful? Physical effort discounting supports common minimization principles in decision making and motor control

When deciding between alternative options, a rational agent chooses on the basis of the desirability of each outcome, including associated costs. As different options typically result in different actions, the effort associated with each action is an essential cost parameter. How do humans discount physical effort when deciding between movements? We used an action-selection task to characterize how subjective effort depends on the parameters of arm transport movements and controlled for potential confounding factors such as delay discounting and performance. First, by repeatedly asking subjects to choose between 2 arm movements of different amplitudes or durations, performed against different levels of force, we identified parameter combinations that subjects experienced as identical in effort (isoeffort curves). Movements with a long duration were judged more effortful than short-duration movements against the same force, while movement amplitudes did not influence effort. Biomechanics of the movements also affected effort, as movements towards the body midline were preferred to movements away from it. Second, by introducing movement repetitions, we further determined that the cost function for choosing between effortful movements had a quadratic relationship with force, while choices were made on the basis of the logarithm of these costs. Our results show that effort-based action selection during reaching cannot easily be explained by metabolic costs. Instead, force-loaded reaches, a widely occurring natural behavior, imposed an effort cost for decision making similar to cost functions in motor control. Our results thereby support the idea that motor control and economic choice are governed by partly overlapping optimization principles.

Economic choice in humans and animals can be understood as a weighing of benefits (e.g., reward) against costs (e.g., effort, delay, risk), leading to a preference for the behavioral option with highest expected utility. The costs of the action associated with a choice can thereby affect its utility: for equivalent benefits, an action that requires less physical effort will be preferred to a more effortful one. Here, we characterized how human subjects assess physical effort when choosing between arm movements. We show that the effort cost of a movement increases with its duration and with the square of the force it is performed against but not with the distance covered. Therefore, the subjective cost that determines decisions does not reflect the objective energetic cost of the actions—i.e., the corresponding metabolic expenditure. Instead, the subjective cost has commonalities with the cost that our central nervous system is believed to minimize for controlling the motor execution of actions. Our findings thus argue in favor of action selection and action control sharing common underlying optimization principles.

A Representation of Effort in Decision-Making and Motor Control

Given two rewarding stimuli, animals tend to choose the more rewarding (or less effortful) option. However, they also move faster toward that stimulus [1-5]. This suggests that reward and effort not only affect decision-making, they also influence motor control [6, 7]. How does the brain compute the effort requirements of a task? Here, we considered data acquired during walking, reaching, flying, or isometric force production. In analyzing the decision-making and motor-control behaviors of various animals, we considered the possibility that the brain may estimate effort objectively, via the metabolic energy consumed to produce the action. We measured the energetic cost of reaching and found that, like walking, it was convex in time, with a global minimum, implying that there existed a movement speed that minimized effort. However, reward made it worthwhile to be energetically inefficient. Using a framework in which utility of an action depended on reward and energetic cost, both discounted in time, we found that it was possible to account for a body of data in which animals were free to choose how to move (reach slow or fast), as well as what to do (walk or fly, produce force F1 or F2). We suggest that some forms of decision-making and motor control may share a common utility in which the brain represents the effort associated with performing an action objectively via its metabolic energy cost and then, like reward, temporally discounts it as a function of movement duration.

Rate-distortion theory and human perception

The fundamental goal of perception is to aid in the achievement of behavioral objectives. This requires extracting and communicating useful information from noisy and uncertain sensory signals. At the same time, given the complexity of sensory information and the limitations of biological information processing, it is necessary that some information must be lost or discarded in the act of perception. Under these circumstances, what constitutes an 'optimal' perceptual system? This paper describes the mathematical framework of rate-distortion theory as the optimal solution to the problem of minimizing the costs of perceptual error subject to strong constraints on the ability to communicate or transmit information. Rate-distortion theory offers a general and principled theoretical framework for developing computational-level models of human perception (Marr, 1982). Models developed in this framework are capable of producing quantitatively precise explanations for human perceptual performance, while yielding new insights regarding the nature and goals of perception. This paper demonstrates the application of rate-distortion theory to two benchmark domains where capacity limits are especially salient in human perception: discrete categorization of stimuli (also known as absolute identification) and visual working memory. A software package written for the R statistical programming language is described that aids in the development of models based on rate-distortion theory.

Do Cost Functions for Tracking Error Generalize across Tasks with Different Noise Levels?

Control of human-machine interfaces are well modeled by computational control models, which take into account the behavioral decisions people make in estimating task dynamics and state for a given control law. This control law is optimized according to a cost function, which for the sake of mathematical tractability is typically represented as a series of quadratic terms. Recent studies have found that people actually use cost functions for reaching tasks that are slightly different than a quadratic function, but it is unclear which of several cost functions best explain human behavior and if these cost functions generalize across tasks of similar nature but different scale. In this study, we used an inverse-decision-theory technique to reconstruct the cost function from empirical data collected on 24 able-bodied subjects controlling a myoelectric interface. Compared with previous studies, this experimental paradigm involved a different control source (myoelectric control, which has inherently large multiplicative noise), a different control interface (control signal was mapped to cursor velocity), and a different task (the tracking position dynamically moved on the screen throughout each trial). Several cost functions, including a linear-quadratic; an inverted Gaussian, and a power function, accurately described the behavior of subjects throughout this experiment better than a quadratic cost function or other explored candidate cost functions (p<0.05). Importantly, despite the differences in the experimental paradigm and a substantially larger scale of error, we found only one candidate cost function whose parameter was consistent with the previous studies: a power function (cost ∝ error^α) with a parameter value of α = 1.69 (1.53–1.78 interquartile range). This result suggests that a power-function is a representative function of user’s error cost over a range of noise amplitudes for pointing and tracking tasks.

Disrupting the supplementary motor area makes physical effort appear less effortful

The perception of physical effort is relatively unaffected by the suppression of sensory afferences, indicating that this function relies mostly on the processing of the central motor command. Neural signals in the supplementary motor area (SMA) correlate with the intensity of effort, suggesting that the motor signal involved in effort perception could originate from this area, but experimental evidence supporting this view is still lacking. Here, we tested this hypothesis by disrupting neural activity in SMA, in primary motor cortex (M1), or in a control site by means of continuous theta-burst transcranial magnetic stimulation, while measuring effort perception during grip forces of different intensities. After each grip force exertion, participants had the opportunity to either accept or refuse to replicate the same effort for varying amounts of reward. In addition to the subjective rating of perceived exertion, effort perception was estimated on the basis of the acceptance rate, the effort replication accuracy, the influence of the effort exerted in trial t on trial t+1, and pupil dilation. We found that disruption of SMA activity, but not of M1, led to a consistent decrease in effort perception, whatever the measure used to assess it. Accordingly, we modeled effort perception in a structural equation model and found that only SMA disruption led to a significant alteration of effort perception. These findings indicate that effort perception relies on the processing of a signal originating from motor-related neural circuits upstream of M1 and that SMA is a key node of this network.

The close proximity of threat: altered distance perception in the anticipation of pain

Pain is an experience that powerfully influences the way we interact with our environment. What is less clear is the influence that pain has on the way we perceive our environment. We investigated the effect that the anticipation of experimental pain (THREAT) and its relief (RELIEF) has on the visual perception of space. Eighteen (11F) healthy volunteers estimated the distance to alternating THREAT and RELIEF stimuli that were placed within reachable space. The results determined that the estimated distance to the THREAT stimulus was significantly underestimated in comparison to the RELIEF stimulus. We conclude that pain-evoking stimuli are perceived as closer to the body than otherwise identical pain-relieving stimuli, an important consideration when applied to our decisions and behaviors in relation to the experience of pain.

Why don't most runners get knee osteoarthritis? A case for per-unit-distance loads

Peak knee joint contact forces ("loads") in running are much higher than they are in walking, where the peak load has been associated with the initiation and progression of knee osteoarthritis. However, runners do not have an especially high risk of osteoarthritis compared with nonrunners. This paradox suggests that running somehow blunts the effect of very high peak joint contact forces, perhaps to provide a load per unit distance (PUD) traveled that is relatively low.

PURPOSE

This study aimed to compare peak and PUD knee joint loads between human walking and running.

METHODS

Fourteen healthy adults walked and ran at self-selected speeds. Ground reaction force and motion capture data were measured and combined with inverse dynamics and musculoskeletal modeling to estimate the peak knee joint loads, PUD knee joint loads, and the impulse of the knee joint contact force for each gait with a matched-pair (within-subject) design.

RESULTS

The peak load was three times higher in running (8.02 vs 2.72 body weight, P < 0.001), but the PUD load did not differ between running and walking (0.80 vs 0.75 body weight per meter, P = 0.098). The impulse of the joint contact force was greater for running than for walking (1.30 vs 1.04 body weight per second, P < 0.001). The peak load increased with increasing running speed, whereas the PUD load decreased with increasing speed.

CONCLUSIONS

Compared with walking, the relatively short duration of ground contact and relatively long length of strides in running seem to blunt the effect of high peak joint loads, such that the PUD loads are no higher than that in walking. Waveform features other than or in addition to the peak value should be considered when studying joint loading and injuries.

Limited encoding of effort by dopamine neurons in a cost-benefit trade-off task

Animals are thought to evaluate the desirability of action options using a unified scale that combines predicted benefits ("rewards"), costs, and the animal's internal motivational state. Midbrain dopamine neurons have long been associated with the reward part of this equation, but it is unclear whether these neurons also estimate the costs of taking an action. We studied the spiking activity of dopamine neurons in the substantia nigra pars compacta of monkeys (Macaca mulatta) during a reaching task in which the energetic costs incurred (friction loads) and the benefits gained (drops of food) were manipulated independently. Although the majority of dopamine neurons encoded the upcoming reward alone, a subset predicted net utility of a course of action by signaling the expected reward magnitude discounted by the invested cost in terms of physical effort. In addition, the tonic activity of some dopamine neurons was slowly reduced in conjunction with the accumulated trials, which is consistent with the hypothesized role for tonic dopamine in the invigoration or motivation of instrumental responding. The present results shed light on an often-hypothesized role for dopamine in the regulation of the balance in natural behaviors between the energy expended and the benefits gained, which could explain why dopamine disorders, such as Parkinson's disease, lead to a breakdown of that balance.

An examination of the generalizability of motor costs

Most approaches to understanding human motor control assume that people maximize their rewards while minimizing their motor efforts. This tradeoff between potential rewards and a sense of effort is quantified with a cost function. While the rewards can change across tasks, our sense of effort is assumed to remain constant and characterize how the nervous system organizes motor control. As such, when a proposed cost function compares well with data it is argued to be the underlying cause of a motor behavior, and not simply a fit to the data. Implicit in this proposition is the assumption that this cost function can then predict new motor behaviors. Here we examined this idea and asked whether an inferred cost function in one setting could explain subject’s behavior in settings that differed dynamically but had identical rewards. We found that the pattern of behavior observed across settings was similar to our predictions of optimal behavior. However, we could not conclude that this behavior was consistent with a conserved sense of effort. These results suggest that the standard forms for quantifying cost may not be sufficient to accurately examine whether or not human motor behavior abides by optimality principles.

Evidence for composite cost functions in arm movement planning: an inverse optimal control approach

An important issue in motor control is understanding the basic principles underlying the accomplishment of natural movements. According to optimal control theory, the problem can be stated in these terms: what cost function do we optimize to coordinate the many more degrees of freedom than necessary to fulfill a specific motor goal? This question has not received a final answer yet, since what is optimized partly depends on the requirements of the task. Many cost functions were proposed in the past, and most of them were found to be in agreement with experimental data. Therefore, the actual principles on which the brain relies to achieve a certain motor behavior are still unclear. Existing results might suggest that movements are not the results of the minimization of single but rather of composite cost functions. In order to better clarify this last point, we consider an innovative experimental paradigm characterized by arm reaching with target redundancy. Within this framework, we make use of an inverse optimal control technique to automatically infer the (combination of) optimality criteria that best fit the experimental data. Results show that the subjects exhibited a consistent behavior during each experimental condition, even though the target point was not prescribed in advance. Inverse and direct optimal control together reveal that the average arm trajectories were best replicated when optimizing the combination of two cost functions, nominally a mix between the absolute work of torques and the integrated squared joint acceleration. Our results thus support the cost combination hypothesis and demonstrate that the recorded movements were closely linked to the combination of two complementary functions related to mechanical energy expenditure and joint-level smoothness.

To reach an object, the brain has to select among a set of possible arm trajectories that displace the hand from an initial to a final desired position. Because of the intrinsic redundancy characterizing the human arm, the number of admissible joint trajectories toward the goal is generally infinite. However, many studies have demonstrated that the range of actual trajectories can be limited to those that result from the fulfillment of some optimal rules. Various cost functions were shown to be relevant in the literature. A peculiar aspect of most of these costs is that each one of them aims at optimizing one specific feature of the movement. The necessary motor flexibility of everyday life, however, might rely on the combination of such cost functions rather than on a single one. Testing this cost combination hypothesis has never been attempted. To this aim we propose a reaching task involving target redundancy to facilitate the comparisons of different candidate costs and to identify the best-fitting one (possibly composite). Using a numerical inverse optimal control method, we show that most participants produced movements corresponding to a strict combination of two subjective costs linked to the mechanical energy consumption and the joint-level smoothness.

Bayesian cue integration as a developmental outcome of reward mediated learning

Average human behavior in cue combination tasks is well predicted by bayesian inference models. As this capability is acquired over developmental timescales, the question arises, how it is learned. Here we investigated whether reward dependent learning, that is well established at the computational, behavioral, and neuronal levels, could contribute to this development. It is shown that a model free reinforcement learning algorithm can indeed learn to do cue integration, i.e. weight uncertain cues according to their respective reliabilities and even do so if reliabilities are changing. We also consider the case of causal inference where multimodal signals can originate from one or multiple separate objects and should not always be integrated. In this case, the learner is shown to develop a behavior that is closest to bayesian model averaging. We conclude that reward mediated learning could be a driving force for the development of cue integration and causal inference.

Risk-sensitivity and the mean-variance trade-off: decision making in sensorimotor control

Numerous psychophysical studies suggest that the sensorimotor system chooses actions that optimize the average cost associated with a movement. Recently, however, violations of this hypothesis have been reported in line with economic theories of decision-making that not only consider the mean payoff, but are also sensitive to risk, that is the variability of the payoff. Here, we examine the hypothesis that risk-sensitivity in sensorimotor control arises as a mean-variance trade-off in movement costs. We designed a motor task in which participants could choose between a sure motor action that resulted in a fixed amount of effort and a risky motor action that resulted in a variable amount of effort that could be either lower or higher than the fixed effort. By changing the mean effort of the risky action while experimentally fixing its variance, we determined indifference points at which participants chose equiprobably between the sure, fixed amount of effort option and the risky, variable effort option. Depending on whether participants accepted a variable effort with a mean that was higher, lower or equal to the fixed effort, they could be classified as risk-seeking, risk-averse or risk-neutral. Most subjects were risk-sensitive in our task consistent with a mean-variance trade-off in effort, thereby, underlining the importance of risk-sensitivity in computational models of sensorimotor control.

Observing the observer (I): meta-bayesian models of learning and decision-making

In this paper, we present a generic approach that can be used to infer how subjects make optimal decisions under uncertainty. This approach induces a distinction between a subject's perceptual model, which underlies the representation of a hidden “state of affairs” and a response model, which predicts the ensuing behavioural (or neurophysiological) responses to those inputs. We start with the premise that subjects continuously update a probabilistic representation of the causes of their sensory inputs to optimise their behaviour. In addition, subjects have preferences or goals that guide decisions about actions given the above uncertain representation of these hidden causes or state of affairs. From a Bayesian decision theoretic perspective, uncertain representations are so-called “posterior” beliefs, which are influenced by subjective “prior” beliefs. Preferences and goals are encoded through a “loss” (or “utility”) function, which measures the cost incurred by making any admissible decision for any given (hidden) state of affair. By assuming that subjects make optimal decisions on the basis of updated (posterior) beliefs and utility (loss) functions, one can evaluate the likelihood of observed behaviour. Critically, this enables one to “observe the observer”, i.e. identify (context- or subject-dependent) prior beliefs and utility-functions using psychophysical or neurophysiological measures. In this paper, we describe the main theoretical components of this meta-Bayesian approach (i.e. a Bayesian treatment of Bayesian decision theoretic predictions). In a companion paper (‘Observing the observer (II): deciding when to decide’), we describe a concrete implementation of it and demonstrate its utility by applying it to simulated and real reaction time data from an associative learning task.

Optimal control predicts human performance on objects with internal degrees of freedom

On a daily basis, humans interact with a vast range of objects and tools. A class of tasks, which can pose a serious challenge to our motor skills, are those that involve manipulating objects with internal degrees of freedom, such as when folding laundry or using a lasso. Here, we use the framework of optimal feedback control to make predictions of how humans should interact with such objects. We confirm the predictions experimentally in a two-dimensional object manipulation task, in which subjects learned to control six different objects with complex dynamics. We show that the non-intuitive behavior observed when controlling objects with internal degrees of freedom can be accounted for by a simple cost function representing a trade-off between effort and accuracy. In addition to using a simple linear, point-mass optimal control model, we also used an optimal control model, which considers the non-linear dynamics of the human arm. We find that the more realistic optimal control model captures aspects of the data that cannot be accounted for by the linear model or other previous theories of motor control. The results suggest that our everyday interactions with objects can be understood by optimality principles and advocate the use of more realistic optimal control models for the study of human motor neuroscience.

Humans are highly skilled at tool use. Simple tools have no internal degrees of freedom. For example, knowing the position and orientation of a hammer allows us to, in theory, predict the forces it will generate on our hand when we wield it and the consequences our actions will have on the hammer. In contrast, more complex tools can have internal degrees of freedom, such as a glass of water in which the motion of the fluid (the internal degree of freedom) is not fully determined by the current position and orientation of the glass. Such objects can be difficult to control. Here we use a robotic interface to simulate complex objects with internal degrees of freedom and find that subjects are able to learn to control the objects and that the pattern of movement found across subjects is similar. We develop an optimal feedback control model and explain complex object interactions as a simple trade-off between effort and task accuracy.

A modular planar robotic manipulandum with end-point torque control

Robotic manipulanda are extensively used in investigation of the motor control of human arm movements. They permit the application of translational forces to the arm based on its state and can be used to probe issues ranging from mechanisms of neural control to biomechanics. However, most current designs are optimized for studying either motor learning or stiffness. Even fewer include end-point torque control which is important for the simulation of objects and the study of tool use. Here we describe a modular, general purpose, two-dimensional planar manipulandum (vBOT) primarily optimized for dynamic learning paradigms. It employs a carbon fibre arm arranged as a parallelogram which is driven by motors via timing pulleys. The design minimizes the intrinsic dynamics of the manipulandum without active compensation. A novel variant of the design (WristBOT) can apply torques at the handle using an add-on cable drive mechanism. In a second variant (StiffBOT) a more rigid arm can be substituted and zero backlash belts can be used, making the StiffBOT more suitable for the study of stiffness. The three variants can be used with custom built display rigs, mounting, and air tables. We investigated the performance of the vBOT and its variants in terms of effective end-point mass, viscosity and stiffness. Finally we present an object manipulation task using the WristBOT. This demonstrates that subjects can perceive the orientation of the principal axis of an object based on haptic feedback arising from its rotational dynamics.

Robotic neurorehabilitation: a computational motor learning perspective

Conventional neurorehabilitation appears to have little impact on impairment over and above that of spontaneous biological recovery. Robotic neurorehabilitation has the potential for a greater impact on impairment due to easy deployment, its applicability across of a wide range of motor impairment, its high measurement reliability, and the capacity to deliver high dosage and high intensity training protocols. We first describe current knowledge of the natural history of arm recovery after stroke and of outcome prediction in individual patients. Rehabilitation strategies and outcome measures for impairment versus function are compared. The topics of dosage, intensity, and time of rehabilitation are then discussed. Robots are particularly suitable for both rigorous testing and application of motor learning principles to neurorehabilitation. Computational motor control and learning principles derived from studies in healthy subjects are introduced in the context of robotic neurorehabilitation. Particular attention is paid to the idea of context, task generalization and training schedule. The assumptions that underlie the choice of both movement trajectory programmed into the robot and the degree of active participation required by subjects are examined. We consider rehabilitation as a general learning problem, and examine it from the perspective of theoretical learning frameworks such as supervised and unsupervised learning. We discuss the limitations of current robotic neurorehabilitation paradigms and suggest new research directions from the perspective of computational motor learning.

Changes in control of saccades during gain adaptation

In a typical short-term saccadic adaptation protocol, the target moves intrasaccadically either toward (gain-down) or away (gain-up) from initial fixation, causing the saccade to complete with an endpoint error. A central question is how the motor system adapts in response to this error: are the motor commands changed to bring the eyes to a different goal, akin to a remapping of the target, or is adaptation focused on the processes that monitor the ongoing motor commands and correct them midflight, akin to changes that act via internal feedback? Here, we found that, in the gain-down paradigm, the brain learned to produce a smaller amplitude saccade by altering the trajectory of the saccade. The adapted saccades had reduced peak velocities, reduced accelerations, shallower decelerations, and increased durations compared with a control saccade of equal amplitude. These changes were consistent with a change in an internal feedback that acted as a forward model. However, in the gain-up paradigm, the brain learned to produce a larger amplitude saccade with trajectories that were identical with those of control saccades of equal amplitude. Therefore, whereas the gain-down paradigm appeared to induce adaptation via an internal feedback that controlled saccades midflight, the gain-up paradigm induced adaptation primarily via target remapping. Our simulations explained that, for each condition, the specific adaptation produced a saccade that brought the eyes to the target with the smallest motor costs.

Composition and decomposition in bimanual dynamic learning

Our ability to skillfully manipulate an object often involves the motor system learning to compensate for the dynamics of the object. When the two arms learn to manipulate a single object they can act cooperatively, whereas when they manipulate separate objects they control each object independently. We examined how learning transfers between these two bimanual contexts by applying force fields to the arms. In a coupled context, a single dynamic is shared between the arms, and in an uncoupled context separate dynamics are experienced independently by each arm. In a composition experiment, we found that when subjects had learned uncoupled force fields they were able to transfer to a coupled field that was the sum of the two fields. However, the contribution of each arm repartitioned over time so that, when they returned to the uncoupled fields, the error initially increased but rapidly reverted to the previous level. In a decomposition experiment, after subjects learned a coupled field, their error increased when exposed to uncoupled fields that were orthogonal components of the coupled field. However, when the coupled field was reintroduced, subjects rapidly readapted. These results suggest that the representations of dynamics for uncoupled and coupled contexts are partially independent. We found additional support for this hypothesis by showing significant learning of opposing curl fields when the context, coupled versus uncoupled, was alternated with the curl field direction. These results suggest that the motor system is able to use partially separate representations for dynamics of the two arms acting on a single object and two arms acting on separate objects.

Flexible representations of dynamics are used in object manipulation

To manipulate an object skillfully, the brain must learn its dynamics, specifying the mapping between applied force and motion. A fundamental issue in sensorimotor control is whether such dynamics are represented in an extrinsic frame of reference tied to the object or an intrinsic frame of reference linked to the arm. Although previous studies have suggested that objects are represented in arm-centered coordinates [1–6], all of these studies have used objects with unusual and complex dynamics. Thus, it is not known how objects with natural dynamics are represented. Here we show that objects with simple (or familiar) dynamics and those with complex (or unfamiliar) dynamics are represented in object- and arm-centered coordinates, respectively. We also show that objects with simple dynamics are represented with an intermediate coordinate frame when vision of the object is removed. These results indicate that object dynamics can be flexibly represented in different coordinate frames by the brain. We suggest that with experience, the representation of the dynamics of a manipulated object may shift from a coordinate frame tied to the arm toward one that is linked to the object. The additional complexity required to represent dynamics in object-centered coordinates would be economical for familiar objects because such a representation allows object use regardless of the orientation of the object in hand.

A computational neuroanatomy for motor control

The study of patients to infer normal brain function has a long tradition in neurology and psychology. More recently, the motor system has been subject to quantitative and computational characterization. The purpose of this review is to argue that the lesion approach and theoretical motor control can mutually inform each other. Specifically, one may identify distinct motor control processes from computational models and map them onto specific deficits in patients. Here we review some of the impairments in motor control, motor learning and higher-order motor control in patients with lesions of the corticospinal tract, the cerebellum, parietal cortex, the basal ganglia, and the medial temporal lobe. We attempt to explain some of these impairments in terms of computational ideas such as state estimation, optimization, prediction, cost, and reward. We suggest that a function of the cerebellum is system identification: to build internal models that predict sensory outcome of motor commands and correct motor commands through internal feedback. A function of the parietal cortex is state estimation: to integrate the predicted proprioceptive and visual outcomes with sensory feedback to form a belief about how the commands affected the states of the body and the environment. A function of basal ganglia is related to optimal control: learning costs and rewards associated with sensory states and estimating the "cost-to-go" during execution of a motor task. Finally, functions of the primary and the premotor cortices are related to implementing the optimal control policy by transforming beliefs about proprioceptive and visual states, respectively, into motor commands.

Social coordination dynamics: measuring human bonding

Spontaneous social coordination has been extensively described in natural settings but so far no controlled methodological approaches have been employed that systematically advance investigations into the possible self-organized nature of bond formation and dissolution between humans. We hypothesized that, under certain contexts, spontaneous synchrony-a well-described phenomenon in biological and physical settings-could emerge spontaneously between humans as a result of information exchange. Here, a new way to quantify interpersonal interactions in real time is proposed. In a simple experimental paradigm, pairs of participants facing each other were required to actively produce actions, while provided (or not) with the vision of similar actions being performed by someone else. New indices of interpersonal coordination, inspired by the theoretical framework of coordination dynamics (based on relative phase and frequency overlap between movements of individuals forming a pair) were developed and used. Results revealed that spontaneous phase synchrony (i.e., unintentional in-phase coordinated behavior) between two people emerges as soon as they exchange visual information, even if they are not explicitly instructed to coordinate with each other. Using the same tools, we also quantified the degree to which the behavior of each individual remained influenced by the social encounter even after information exchange had been removed, apparently a kind of social memory.

Decision theory: what "should" the nervous system do?

The purpose of our nervous system is to allow us to successfully interact with our environment. This normative idea is formalized by decision theory that defines which choices would be most beneficial. We live in an uncertain world, and each decision may have many possible outcomes; choosing the best decision is thus complicated. Bayesian decision theory formalizes these problems in the presence of uncertainty and often provides compact models that predict observed behavior. With its elegant formalization of the problems faced by the nervous system, it promises to become a major inspiration for studies in neuroscience.

Simultaneous bimanual dynamics are learned without interference

Dynamic learning in humans has been extensively studied using externally applied force fields to perturb movements of the arm. These studies have focused on unimanual learning in which a force field is applied to only one arm. Here we examine dynamic learning during bimanual movements. Specifically we examine learning of a force field in one arm when the other arm makes movements in a null field or in a force field. For both the dominant and non-dominant arms, the learning (change in performance over the exposure period) was the same regardless of whether the other arm moved in a force field, equivalent either in intrinsic or extrinsic coordinates, or moved in a null field. Moreover there were no significant differences in learning in these bimanual tasks compared to unimanual learning, when one arm experienced a force field and the other arm was at rest. Although the learning was the same, there was an overall increase in error for the non-dominant arm for all bimanual conditions compared to the unimanual condition. This increase in error was the result of bimanual movement alone and was present even in the initial training phase before any forces were introduced. We conclude that, during bimanual movements, the application of a force field to one arm neither interferes with nor facilitates simultaneous learning of a force field applied to the other arm.

Why don't we move faster? Parkinson's disease, movement vigor, and implicit motivation

People generally select a similar speed for a given motor task, such as reaching for a cup. One well established determinant of movement time is the speed-accuracy trade-off: movement time increases with the accuracy requirement. A second possible determinant is the energetic cost of making a movement. Parkinson's disease (PD), a condition characterized by generalized movement slowing (bradykinesia), provides the opportunity to directly explore this second possibility. We compared reaching movements of patients with PD with those of control subjects in a speed-accuracy trade-off task comprising conditions of increasing difficulty. Subjects completed as many trials as necessary to make 20 movements within a required speed range (trials to criterion, N(c)). Difficulty was reflected in endpoint accuracy and N(c). Patients were as accurate as control subjects in all conditions (i.e., PD did not affect the speed-accuracy trade-off). However, N(c) was consistently higher in patients, indicating reluctance to move fast although accuracy was not compromised. Specifically, the dependence of N(c) on movement energy cost (slope S(N)) was steeper in patients than in control subjects. This difference in S(N) suggests that bradykinesia represents an implicit decision not to move fast because of a shift in the cost/benefit ratio of the energy expenditure needed to move at normal speed. S(N) was less steep, but statistically significant, in control subjects, which demonstrates a role for energetic cost in the normal control of movement speed. We propose that, analogous to the established role of dopamine in explicit reward-seeking behavior, the dopaminergic projection to the striatum provides a signal for implicit "motor motivation."

Actions and consequences in bimanual interaction are represented in different coordinate systems

Moving one part of the body can generate interaction forces that tend to destabilize other parts of the body. However, stability is maintained by mechanisms that predict and actively oppose these interaction forces. When our body or environment changes, these anticipatory mechanisms adapt so as to continue to produce accurate predictions. In this study, we examine the acquisition of a novel predictive coordination between the arms, in a situation in which a force is produced on one hand as a consequence of the action of the other hand. Specifically, a force was applied to the left hand that depended on the velocity of the right hand. With practice, subjects learned to stabilize the perturbed left arm during right-arm movements by predicting and actively opposing the externally applied forces. After adaptation, we examined how learning generalized to a new joint configuration of the right or left arm to investigate the coordinate systems in which the internal transformation from movement to force is represented. This revealed a dissociation between the representation of the action of the right arm and the representation of its consequence, that is the force on the left arm. The movement is represented in extrinsic coordinates related to the velocity of the hand, whereas the force resulting from the movement is represented in a joint-based intrinsic coordinate system.

Bayesian decision theory in sensorimotor control

Action selection is a fundamental decision process for us, and depends on the state of both our body and the environment. Because signals in our sensory and motor systems are corrupted by variability or noise, the nervous system needs to estimate these states. To select an optimal action these state estimates need to be combined with knowledge of the potential costs or rewards of different action outcomes. We review recent studies that have investigated the mechanisms used by the nervous system to solve such estimation and decision problems, which show that human behaviour is close to that predicted by Bayesian Decision Theory. This theory defines optimal behaviour in a world characterized by uncertainty, and provides a coherent way of describing sensorimotor processes.