When money is not enough: awareness, success, and variability in motor learning

Motor competence is related to acquisition of error-based but not reinforcement learning in children ages 6 to 12

Background

An essential component of childhood development is increasing motor competence. Poor motor learning is often thought to underlie impaired motor competence, but this link is unclear in previous studies.

Aims

Our aim was to test the relationship between motor competence and motor learning in the acquisition phase. Both reinforcement learning (RL) and error-based learning (EBL) were tested. We hypothesized that slower RL and slower EBL acquisition rates would relate to lower motor competence.

Methods and procedures

Eighty-six participants ages 6-12 performed a target throwing task under RL and EBL conditions. The Movement Assessment Battery for Children - 2nd edition (MABC-2) provided a measure of motor competence. We assessed EBL and RL acquisition rates, baseline variability, and baseline bias from the throwing task.

Outcomes and results

In a multiple linear regression model, baseline variability (β = -0.49, p = <0.001) and the EBL acquisition rate (β = -0.24, p = 0.018) significantly explained the MABC-2 score. Participants with higher baseline variability and slower EBL acquisition had lower motor competence scores. The RL acquisition rate was independent of MABC-2 score suggesting that RL may be less of a contributor to poor motor competence.

Conclusions and implications

Children with slower EBL acquisition had lower motor competence scores but RL acquisition was unrelated to the level of motor competence. Emphasizing the unrelated reinforcement mechanisms over error-based mechanisms during motor skill interventions may help children with poor motor competence better acquire new motor skills.

The effects of reward and punishment on the performance of ping-pong ball bouncing

Introduction

Reward and punishment modulate behavior. In real-world motor skill learning, reward and punishment have been found to have dissociable effects on optimizing motor skill learning, but the scientific basis for these effects is largely unknown.

Methods

In the present study, we investigated the effects of reward and punishment on the performance of real-world motor skill learning. Specifically, three groups of participants were trained and tested on a ping-pong ball bouncing task for three consecutive days. The training and testing sessions were identical across the three days: participants were trained with their right (dominant) hand each day under conditions of either reward, punishment, or a neutral control condition (neither). Before and after the training session, all participants were tested with their right and left hands without any feedback.

Results

We found that punishment promoted early learning, while reward promoted late learning. Reward facilitated short-term memory, while punishment impaired long-term memory. Both reward and punishment interfered with long-term memory gains. Interestingly, the effects of reward and punishment transferred to the left hand.

Discussion

The results show that reward and punishment have different effects on real-world motor skill learning. The effects change with training and transfer readily to novel contexts. The results suggest that reward and punishment may act on different learning processes and engage different neural mechanisms during real-world motor skill learning. In addition, high-level metacognitive processes may be enabled by the additional reinforcement feedback during real-world motor skill learning. Our findings provide new insights into the mechanisms underlying motor learning, and may have important implications for practical applications such as sports training and motor rehabilitation.

Reinforcement Learning during Locomotion

When learning a new motor skill, people often must use trial and error to discover which movement is best. In the reinforcement learning framework, this concept is known as exploration and has been linked to increased movement variability in motor tasks. For locomotor tasks, however, increased variability decreases upright stability. As such, exploration during gait may jeopardize balance and safety, making reinforcement learning less effective. Therefore, we set out to determine if humans could acquire and retain a novel locomotor pattern using reinforcement learning alone. Young healthy male and female participants walked on a treadmill and were provided with binary reward feedback (indicated by a green checkmark on the screen) that was tied to a fixed monetary bonus, to learn a novel stepping pattern. We also recruited a comparison group who walked with the same novel stepping pattern but did so by correcting for target error, induced by providing real-time veridical visual feedback of steps and a target. In two experiments, we compared learning, motor variability, and two forms of motor memories between the groups. We found that individuals in the binary reward group did, in fact, acquire the new walking pattern by exploring (increasing motor variability). Additionally, while reinforcement learning did not increase implicit motor memories, it resulted in more accurate explicit motor memories compared with the target error group. Overall, these results demonstrate that humans can acquire new walking patterns with reinforcement learning and retain much of the learning over 24 h.

Interaction between decision-making and motor learning when selecting reach targets in the presence of bias and noise

Motor errors can have both bias and noise components. Bias can be compensated for by adaptation and, in tasks in which the magnitude of noise varies across the environment, noise can be reduced by identifying and then acting in less noisy regions of the environment. Here we examine how these two processes interact when participants reach under a combination of an externally imposed visuomotor bias and noise. In a center-out reaching task, participants experienced noise (zero-mean random visuomotor rotations) that was target-direction dependent with a standard deviation that increased linearly from a least-noisy direction. They also experienced a constant bias, a visuomotor rotation that varied (across groups) from 0 to 40 degrees. Critically, on each trial, participants could select one of three targets to reach to, thereby allowing them to potentially select targets close to the least-noisy direction. The group who experienced no bias (0 degrees) quickly learned to select targets close to the least-noisy direction. However, groups who experienced a bias often failed to identify the least-noisy direction, even though they did partially adapt to the bias. When noise was introduced after participants experienced and adapted to a 40 degrees bias (without noise) in all directions, they exhibited an improved ability to find the least-noisy direction. We developed two models-one for reach adaptation and one for target selection-that could explain participants' adaptation and target-selection behavior. Our data and simulations indicate that there is a trade-off between adaptation and selection. Specifically, because bias learning is local, participants can improve performance, through adaptation, by always selecting targets that are closest to a chosen direction. However, this comes at the expense of improving performance, through selection, by reaching toward targets in different directions to find the least-noisy direction.

The dissociable effects of reward on sequential motor behavior

Reward has consistently been shown to enhance motor behavior; however, its beneficial effects appear to be largely unspecific. For example, reward is associated with both rapid and training-dependent improvements in performance, with a mechanistic account of these effects currently lacking. Here we tested the hypothesis that these distinct reward-based improvements are driven by dissociable reward types: monetary incentive and performance feedback. Whereas performance feedback provides information on how well a motor task has been completed (knowledge of performance), monetary incentive increases the motivation to perform optimally without providing a performance-based learning signal. Experiment 1 showed that groups who received monetary incentive rapidly improved movement times (MTs), using a novel sequential reaching task. In contrast, only groups with correct performance-based feedback showed learning-related improvements. Importantly, pairing both maximized MT performance gains and accelerated movement fusion. Fusion describes an optimization process during which neighboring sequential movements blend together to form singular actions. Results from experiment 2 served as a replication and showed that fusion led to enhanced performance speed while also improving movement efficiency through increased smoothness. Finally, experiment 3 showed that these improvements in performance persist for 24 h even without reward availability. This highlights the dissociable impact of monetary incentive and performance feedback, with their combination maximizing performance gains and leading to stable improvements in the speed and efficiency of sequential actions.NEW & NOTEWORTHY Our work provides a mechanistic framework for how reward influences motor behavior. Specifically, we show that rapid improvements in speed and accuracy are driven by reward presented in the form of money, whereas knowledge of performance through performance feedback leads to training-based improvements. Importantly, combining both maximized performance gains and led to improvements in movement quality through fusion, which describes an optimization process during which sequential movements blend into a single action.

Clustering analysis of movement kinematics in reinforcement learning

Reinforcement learning has been used as an experimental model of motor skill acquisition, where at times movements are successful and thus reinforced. One fundamental problem is to understand how humans select exploration over exploitation during learning. The decision could be influenced by factors such as task demands and reward availability. In this study, we applied a clustering algorithm to examine how a change in the accuracy requirements of a task affected the choice of exploration over exploitation. Participants made reaching movements to an unseen target using a planar robot arm and received reward after each successful movement. For one group of participants, the width of the hidden target decreased after every other training block. For a second group, it remained constant. The clustering algorithm was applied to the kinematic data to characterize motor learning on a trial-to-trial basis as a sequence of movements, each belonging to one of the identified clusters. By the end of learning, movement trajectories across all participants converged primarily to a single cluster with the greatest number of successful trials. Within this analysis framework, we defined exploration and exploitation as types of behavior in which two successive trajectories belong to different or similar clusters, respectively. The frequency of each mode of behavior was evaluated over the course of learning. It was found that by reducing the target width, participants used a greater variety of different clusters and displayed more exploration than exploitation. Excessive exploration relative to exploitation was found to be detrimental to subsequent motor learning.

NEW & NOTEWORTHY The choice of exploration versus exploitation is a fundamental problem in learning new motor skills through reinforcement. In this study, we employed a data-driven approach to characterize movements on a trial-by-trial basis with an unsupervised clustering algorithm. Using this technique, we found that changes in task demands and, in particular, in the required accuracy of movements, influenced the ratio of exploration to exploitation. This analysis framework provides an attractive tool to investigate mechanisms of explorative and exploitative behavior while studying motor learning.

Learning a reach trajectory based on binary reward feedback

Binary reward feedback on movement success is sufficient for learning some simple sensorimotor mappings in a reaching task, but not for some other tasks in which multiple kinematic factors contribute to performance. The critical condition for learning in more complex tasks remains unclear. Here, we investigate whether reward-based motor learning is possible in a multi-dimensional trajectory matching task and whether simplifying the task by providing feedback on one factor at a time ('factorized feedback') can improve learning. In two experiments, participants performed a trajectory matching task in which learning was measured as a reduction in the error. In Experiment 1, participants matched a straight trajectory slanted in depth. We factorized the task by providing feedback on the slant error, the length error, or on their composite. In Experiment 2, participants matched a curved trajectory, also slanted in depth. In this experiment, we factorized the feedback by providing feedback on the slant error, the curvature error, or on the integral difference between the matched and target trajectory. In Experiment 1, there was anecdotal evidence that participants learnt the multidimensional task. Factorization did not improve learning. In Experiment 2, there was anecdotal evidence the multidimensional task could not be learnt. We conclude that, within a complexity range, multiple kinematic factors can be learnt in parallel.

Intention to learn modulates the impact of reward and punishment on sequence learning

In real-world settings, learning is often characterised as intentional: learners are aware of the goal during the learning process, and the goal of learning is readily dissociable from the awareness of what is learned. Recent evidence has shown that reward and punishment (collectively referred to as valenced feedback) are important factors that influence performance during learning. Presently, however, studies investigating the impact of valenced feedback on skill learning have only considered unintentional learning, and therefore the interaction between intentionality and valenced feedback has not been systematically examined. The present study investigated how reward and punishment impact behavioural performance when participants are instructed to learn in a goal-directed fashion (i.e. intentionally) rather than unintentionally. In Experiment 1, participants performed the serial response time task with reward, punishment, or control feedback and were instructed to ignore the presence of the sequence, i.e., learn unintentionally. Experiment 2 followed the same design, but participants were instructed to intentionally learn the sequence. We found that punishment significantly benefitted performance during learning only when participants learned unintentionally, and we observed no effect of punishment when participants learned intentionally. Thus, the impact of feedback on performance may be influenced by goal of the learner.

Quantifying exploration in reward-based motor learning

Exploration in reward-based motor learning is observable in experimental data as increased variability. In order to quantify exploration, we compare three methods for estimating other sources of variability: sensorimotor noise. We use a task in which participants could receive stochastic binary reward feedback following a target-directed weight shift. Participants first performed six baseline blocks without feedback, and next twenty blocks alternating with and without feedback. Variability was assessed based on trial-to-trial changes in movement endpoint. We estimated sensorimotor noise by the median squared trial-to-trial change in movement endpoint for trials in which no exploration is expected. We identified three types of such trials: trials in baseline blocks, trials in the blocks without feedback, and rewarded trials in the blocks with feedback. We estimated exploration by the median squared trial-to-trial change following non-rewarded trials minus sensorimotor noise. As expected, variability was larger following non-rewarded trials than following rewarded trials. This indicates that our reward-based weight-shifting task successfully induced exploration. Most importantly, our three estimates of sensorimotor noise differed: the estimate based on rewarded trials was significantly lower than the estimates based on the two types of trials without feedback. Consequently, the estimates of exploration also differed. We conclude that the quantification of exploration depends critically on the type of trials used to estimate sensorimotor noise. We recommend the use of variability following rewarded trials.

Domain-Specific Working Memory, But Not Dopamine-Related Genetic Variability, Shapes Reward-Based Motor Learning

The addition of rewarding feedback to motor learning tasks has been shown to increase the retention of learning, spurring interest in its possible utility for rehabilitation. However, motor tasks using rewarding feedback have repeatedly been shown to lead to great interindividual variability in performance. Understanding the causes of such variability is vital for maximizing the potential benefits of reward-based motor learning. Thus, using a large human cohort of both sexes (n = 241), we examined whether spatial (SWM), verbal, and mental rotation (RWM) working memory capacity and dopamine-related genetic profiles were associated with performance in two reward-based motor tasks. The first task assessed the participant's ability to follow a slowly shifting reward region based on hit/miss (binary) feedback. The second task investigated the participant's capacity to preserve performance with binary feedback after adapting to the rotation with full visual feedback. Our results demonstrate that higher SWM is associated with greater success and an enhanced capacity to reproduce a successful motor action, measured as change in reach angle following reward. In contrast, higher RWM was predictive of an increased propensity to express an explicit strategy when required to make large reach angle adjustments. Therefore, SWM and RWM were reliable, but dissociable, predictors of success during reward-based motor learning. Change in reach direction following failure was also a strong predictor of success rate, although we observed no consistent relationship with working memory. Surprisingly, no dopamine-related genotypes predicted performance. Therefore, working memory capacity plays a pivotal role in determining individual ability in reward-based motor learning.SIGNIFICANCE STATEMENT Reward-based motor learning tasks have repeatedly been shown to lead to idiosyncratic behaviors that cause varying degrees of task success. Yet, the factors determining an individual's capacity to use reward-based feedback are unclear. Here, we assessed a wide range of possible candidate predictors, and demonstrate that domain-specific working memory plays an essential role in determining individual capacity to use reward-based feedback. Surprisingly, genetic variations in dopamine availability were not found to play a role. This is in stark contrast with seminal work in the reinforcement and decision-making literature, which show strong and replicated effects of the same dopaminergic genes in decision-making. Therefore, our results provide novel insights into reward-based motor learning, highlighting a key role for domain-specific working memory capacity.

Somatosensory working memory in human reinforcement-based motor learning

Recent studies using visuomotor adaptation and sequence learning tasks have assessed the involvement of working memory in the visuospatial domain. The capacity to maintain previously performed movements in working memory is perhaps even more important in reinforcement-based learning to repeat accurate movements and avoid mistakes. Using this kind of task in the present work, we tested the relationship between somatosensory working memory and motor learning. The first experiment involved separate memory and motor learning tasks. In the memory task, the participant's arm was displaced in different directions by a robotic arm, and the participant was asked to judge whether a subsequent test direction was one of the previously presented directions. In the motor learning task, participants made reaching movements to a hidden visual target and were provided with positive feedback as reinforcement when the movement ended in the target zone. It was found that participants that had better somatosensory working memory showed greater motor learning. In a second experiment, we designed a new task in which learning and working memory trials were interleaved, allowing us to study participants' memory for movements they performed as part of learning. As in the first experiment, we found that participants with better somatosensory working memory also learned more. Moreover, memory performance for successful movements was better than for movements that failed to reach the target. These results suggest that somatosensory working memory is involved in reinforcement motor learning and that this memory preferentially keeps track of reinforced movements. NEW & NOTEWORTHY The present work examined somatosensory working memory in reinforcement-based motor learning. Working memory performance was reliably correlated with the extent of learning. With the use of a paradigm in which learning and memory trials were interleaved, memory was assessed for movements performed during learning. Movements that received positive feedback were better remembered than movements that did not. Thus working memory does not track all movements equally but is biased to retain movements that were rewarded.

The relationship between reinforcement and explicit control during visuomotor adaptation

The motor system’s ability to adapt to environmental changes is essential for maintaining accurate movements. Such adaptation recruits several distinct systems: cerebellar sensory-prediction error learning, success-based reinforcement, and explicit control. Although much work has focused on the relationship between cerebellar learning and explicit control, there is little research regarding how reinforcement and explicit control interact. To address this, participants first learnt a 20° visuomotor displacement. After reaching asymptotic performance, binary, hit-or-miss feedback (BF) was introduced either with or without visual feedback, the latter promoting reinforcement. Subsequently, retention was assessed using no-feedback trials, with half of the participants in each group being instructed to stop aiming off target. Although BF led to an increase in retention of the visuomotor displacement, instructing participants to stop re-aiming nullified this effect, suggesting explicit control is critical to BF-based reinforcement. In a second experiment, we prevented the expression or development of explicit control during BF performance, by either constraining participants to a short preparation time (expression) or by introducing the displacement gradually (development). Both manipulations strongly impaired BF performance, suggesting reinforcement requires both recruitment and expression of an explicit component. These results emphasise the pivotal role explicit control plays in reinforcement-based motor learning.

Added value of money on motor performance feedback: Increased left central beta-band power for rewards and fronto-central theta-band power for punishments

Monetary rewards and punishments have been shown to respectively enhance retention of motor memories and short-term motor performance, but their underlying neural bases in the context of motor control tasks remain unclear. Using electroencephalography (EEG), the present study tested the hypothesis that monetary rewards and punishments are respectively reflected in post-feedback beta-band (20-30 Hz) and theta-band (3-8 Hz) oscillatory power. While participants performed upper limb reaching movements toward visual targets using their right hand, the delivery of monetary rewards and punishments was manipulated as well as their probability (i.e., by changing target size). Compared to unrewarded and unpunished trials, monetary rewards and the successful avoidance of punishments both entailed greater beta-band power at left central electrodes overlaying contralateral motor areas. In contrast, monetary punishments and reward omissions both entailed increased theta-band power at fronto-central scalp sites. Additional analyses revealed that beta-band power was further increased when rewards were lowly probable. In light of previous work demonstrating similar beta-band modulations in basal ganglia during reward processing, the present results may reflect functional communication of reward-related information between the basal ganglia and motor cortical regions. In turn, the increase in fronto-central theta-band power after monetary punishments may reflect an emphasized cognitive need for behavioral adjustments. Globally, the present work identifies possible neural substrates for the growing behavioral evidence showing beneficial effects of monetary feedback on motor learning and performance.

Contribution of explicit processes to reinforcement-based motor learning

Despite increasing interest in the role of reward in motor learning, the underlying mechanisms remain ill defined. In particular, the contribution of explicit processes to reward-based motor learning is unclear. To address this, we examined subjects' ( n = 30) ability to learn to compensate for a gradually introduced 25° visuomotor rotation with only reward-based feedback (binary success/failure). Only two-thirds of subjects ( n = 20) were successful at the maximum angle. The remaining subjects initially followed the rotation but after a variable number of trials began to reach at an insufficiently large angle and subsequently returned to near-baseline performance ( n = 10). Furthermore, those who were successful accomplished this via a large explicit component, evidenced by a reduction in reach angle when they were asked to remove any strategy they employed. However, both groups displayed a small degree of remaining retention even after the removal of this explicit component. All subjects made greater and more variable changes in reach angle after incorrect (unrewarded) trials. However, subjects who failed to learn showed decreased sensitivity to errors, even in the initial period in which they followed the rotation, a pattern previously found in parkinsonian patients. In a second experiment, the addition of a secondary mental rotation task completely abolished learning ( n = 10), while a control group replicated the results of the first experiment ( n = 10). These results emphasize a pivotal role of explicit processes during reinforcement-based motor learning, and the susceptibility of this form of learning to disruption has important implications for its potential therapeutic benefits. NEW & NOTEWORTHY We demonstrate that learning a visuomotor rotation with only reward-based feedback is principally accomplished via the development of a large explicit component. Furthermore, this form of learning is susceptible to disruption with a secondary task. The results suggest that future experiments utilizing reward-based feedback should aim to dissect the roles of implicit and explicit reinforcement learning systems. Therapeutic motor learning approaches based on reward should be aware of the sensitivity to disruption.

It's Not (Only) the Mean that Matters: Variability, Noise and Exploration in Skill Learning

Mastering a motor skill is typified by a decrease in variability. However, variability is much more than the undesired signature of discoordination: structure in both its distributional properties and temporal sequence can reveal control priorities. Extending from the notion that signal-dependent noise corrupts information transmission in the neuromotor system, this review tracks more recent recognitions that the complex dynamic motor system in its interaction with task constraints creates high-dimensional spaces with multiple equivalent solutions. Further analysis differentiates these solutions to have different degrees of noise-sensitivity, goal-relevance or additional costs. Practice proceeds from exploration of these solution spaces to then exploitation with further channeling of noise. Extended practice leads to fine-tuning of skill brought about by reducing noise. These distinct changes in variability are suggested as a way to characterize stages of learning. Capitalizing on the sensitivity of the CNS to noise, interventions can add extrinsic or amplify intrinsic noise to guide (re-)learning desired behaviors. The persistence and generalization of acquired skill is still largely understudied, although an essential element of skill. Consistent with advances in the physical sciences, there is increasing realization that noise can have beneficial effects. Analysis of the non-random structure of variability may reveal more than analysis of only its mean.

Role of Constant, Random and Blocked Practice in an Electromyography-Based Oral Motor Learning Task

PURPOSE

The role of principles of motor learning (PMLs) in speech has received much attention in the past decade. Oral motor learning, however, has not received similar consideration. This study evaluated the role of three practice conditions in an oral motor tracking task.

METHOD

Forty-five healthy adult participants were randomly and equally assigned to one of three practice conditions (constant, blocked, and random) and participated in an electromyography-based task. The study consisted of four sessions, at one session a day for four consecutive days. The first three days sessions included a practice phase, with immediate visual feedback, and an immediate retention phase, without visual feedback. The fourth session did not include practice, but only delayed retention testing, lasting 10-15 minutes, without visual feedback.

RESULTS

Random group participants performed better than participants in constant and blocked practice conditions on all the four days. Constant group participants demonstrated superior learning over blocked group participants only on day 4.

CONCLUSION

Findings indicate that random practice facilitates oral motor learning, which is in line with limb/speech motor learning literature. Future research should systematically investigate the outcomes of random practice as a function of different oral and speech-based tasks.

Minimizing endpoint variability through reinforcement learning during reaching movements involving shoulder, elbow and wrist

Reaching movements are comprised of the coordinated action across multiple joints. The human skeleton is redundant for this task because different joint configurations can lead to the same endpoint in space. How do people learn to use combinations of joints that maximize success in goal-directed motor tasks? To answer this question, we used a 3-degree-of-freedom manipulandum to measure shoulder, elbow and wrist joint movements during reaching in a plane. We tested whether a shift in the relative contribution of the wrist and elbow joints to a reaching movement could be learned by an implicit reinforcement regime. Unknown to the participants, we decreased the task success for certain joint configurations (wrist flexion or extension, respectively) by adding random variability to the endpoint feedback. In return, the opposite wrist postures were rewarded in the two experimental groups (flexion and extension group). We found that the joint configuration slowly shifted towards movements that provided more control over the endpoint and hence higher task success. While the overall learning was significant, only the group that was guided to extend the wrist joint more during the movement showed substantial learning. Importantly, all changes in movement pattern occurred independent of conscious awareness of the experimental manipulation. These findings suggest that the motor system is generally sensitive to its output variability and can optimize joint-space solutions that minimize task-relevant output variability. We discuss biomechanical biases (e.g. joint's range of movement) that could impose hurdles to the learning process.

Somatic and Reinforcement-Based Plasticity in the Initial Stages of Human Motor Learning

As one learns to dance or play tennis, the desired somatosensory state is typically unknown. Trial and error is important as motor behavior is shaped by successful and unsuccessful movements. As an experimental model, we designed a task in which human participants make reaching movements to a hidden target and receive positive reinforcement when successful. We identified somatic and reinforcement-based sources of plasticity on the basis of changes in functional connectivity using resting-state fMRI before and after learning. The neuroimaging data revealed reinforcement-related changes in both motor and somatosensory brain areas in which a strengthening of connectivity was related to the amount of positive reinforcement during learning. Areas of prefrontal cortex were similarly altered in relation to reinforcement, with connectivity between sensorimotor areas of putamen and the reward-related ventromedial prefrontal cortex strengthened in relation to the amount of successful feedback received. In other analyses, we assessed connectivity related to changes in movement direction between trials, a type of variability that presumably reflects exploratory strategies during learning. We found that connectivity in a network linking motor and somatosensory cortices increased with trial-to-trial changes in direction. Connectivity varied as well with the change in movement direction following incorrect movements. Here the changes were observed in a somatic memory and decision making network involving ventrolateral prefrontal cortex and second somatosensory cortex. Our results point to the idea that the initial stages of motor learning are not wholly motor but rather involve plasticity in somatic and prefrontal networks related both to reward and exploration.

SIGNIFICANCE STATEMENT

In the initial stages of motor learning, the placement of the limbs is learned primarily through trial and error. In an experimental analog, participants make reaching movements to a hidden target and receive positive feedback when successful. We identified sources of plasticity based on changes in functional connectivity using resting-state fMRI. The main finding is that there is a strengthening of connectivity between reward-related prefrontal areas and sensorimotor areas in the basal ganglia and frontal cortex. There is also a strengthening of connectivity related to movement exploration in sensorimotor circuits involved in somatic memory and decision making. The results indicate that initial stages of motor learning depend on plasticity in somatic and prefrontal networks related to reward and exploration.

Chronic Stroke Survivors Improve Reaching Accuracy by Reducing Movement Variability at the Trained Movement Speed

BACKGROUND

Recovery from stroke is often said to have "plateaued" after 6 to 12 months. Yet training can still improve performance even in the chronic phase. Here we investigate the biomechanics of accuracy improvements during a reaching task and test whether they are affected by the speed at which movements are practiced.

METHOD

We trained 36 chronic stroke survivors (57.5 years, SD ± 11.5; 10 females) over 4 consecutive days to improve endpoint accuracy in an arm-reaching task (420 repetitions/day). Half of the group trained using fast movements and the other half slow movements. The trunk was constrained allowing only shoulder and elbow movement for task performance.

RESULTS

Before training, movements were variable, tended to undershoot the target, and terminated in contralateral workspace (flexion bias). Both groups improved movement accuracy by reducing trial-to-trial variability; however, change in endpoint bias (systematic error) was not significant. Improvements were greatest at the trained movement speed and generalized to other speeds in the fast training group. Small but significant improvements were observed in clinical measures in the fast training group.

CONCLUSIONS

The reduction in trial-to-trial variability without an alteration to endpoint bias suggests that improvements are achieved by better control over motor commands within the existing repertoire. Thus, 4 days' training allows stroke survivors to improve movements that they can already make. Whether new movement patterns can be acquired in the chronic phase will need to be tested in longer term studies. We recommend that training needs to be performed at slow and fast movement speeds to enhance generalization.

Using noise to shape motor learning

Each of our movements is selected from any number of alternative movements. Some studies have shown evidence that the central nervous system (CNS) chooses to make the specific movements that are least affected by motor noise. Previous results showing that the CNS has a natural tendency to minimize the effects of noise make the direct prediction that if the relationship between movements and noise were to change, the specific movements people learn to make would also change in a predictable manner. Indeed, this has been shown for well-practiced movements such as reaching. Here, we artificially manipulated the relationship between movements and visuomotor noise by adding noise to a motor task in a novel redundant geometry such that there arose a single control policy that minimized the noise. This allowed us to see whether, for a novel motor task, people could learn the specific control policy that minimized noise or would need to employ other compensation strategies to overcome the added noise. As predicted, subjects were able to learn movements that were biased toward the specific ones that minimized the noise, suggesting not only that the CNS can learn to minimize the effects of noise in a novel motor task but also that artificial visuomotor noise can be a useful tool for teaching people to make specific movements. Using noise as a teaching signal promises to be useful for rehabilitative therapies and movement training with human-machine interfaces.

NEW & NOTEWORTHY

Many theories argue that we choose to make the specific movements that minimize motor noise. Here, by changing the relationship between movements and noise, we show that people actively learn to make movements that minimize noise. This not only provides direct evidence for the theories of noise minimization but presents a way to use noise to teach specific movements to improve rehabilitative therapies and human-machine interface control.

Increased Adaptation Rates and Reduction in Trial-by-Trial Variability in Subjects with Cerebral Palsy Following a Multi-session Locomotor Adaptation Training

Cerebral Palsy (CP) results from an insult to the developing brain and is associated with deficits in locomotor and manual skills and in sensorimotor adaptation. We hypothesized that the poor sensorimotor adaptation in persons with CP is related to their high execution variability and does not reflect a general impairment in adaptation learning. We studied the interaction between performance variability and adaptation deficits using a multi-session locomotor adaptation design in persons with CP. Six adolescents with diplegic CP were exposed, during a period of 15 weeks, to a repeated split-belt treadmill perturbation spread over 30 sessions and were tested again 6 months after the end of training. Compared to age-matched healthy controls, subjects with CP showed poor adaptation and high execution variability in the first exposure to the perturbation. Following training they showed marked reduction in execution variability and an increase in learning rates. The reduction in variability and the improvement in adaptation were highly correlated in the CP group and were retained 6 months after training. Interestingly, despite reducing their variability in the washout phase, subjects with CP did not improve learning rates during washout phases that were introduced only four times during the experiment. Our results suggest that locomotor adaptation in subjects with CP is related to their execution variability. Nevertheless, while variability reduction is generalized to other locomotor contexts, the development of savings requires both reduction in execution variability and multiple exposures to the perturbation.

Rewarding imperfect motor performance reduces adaptive changes

Could a pat on the back affect motor adaptation? Recent studies indeed suggest that rewards can boost motor adaptation. However, the rewards used were typically reward gradients that carried quite detailed information about performance. We investigated whether simple binary rewards affected how participants learned to correct for a visual rotation of performance feedback in a 3D pointing task. To do so, we asked participants to align their unseen hand with virtual target cubes in alternating blocks with and without spatial performance feedback. Forty participants were assigned to one of two groups: a ‘spatial only’ group, in which the feedback consisted of showing the (perturbed) endpoint of the hand, or to a ‘spatial & reward’ group, in which a reward could be received in addition to the spatial feedback. In addition, six participants were tested in a ‘reward only’ group. Binary reward was given when the participants’ hand landed in a virtual ‘hit area’ that was adapted to individual performance to reward about half the trials. The results show a typical pattern of adaptation in both the ‘spatial only’ and the ‘spatial & reward’ groups, whereas the ‘reward only’ group was unable to adapt. The rewards did not affect the overall pattern of adaptation in the ‘spatial & reward’ group. However, on a trial-by-trial basis, the rewards reduced adaptive changes to spatial errors.

Formation of a long-term memory for visuomotor adaptation following only a few trials of practice

The term savings refers to faster motor adaptation upon reexposure to a previously experienced perturbation, a phenomenon thought to reflect the existence of a long-term motor memory. It is commonly assumed that sustained practice during the first perturbation exposure is necessary to create this memory. Here we sought to test this assumption by determining the minimum amount of experience necessary during initial adaptation to a visuomotor rotation to bring about savings the following day. Four groups of human subjects experienced 2, 5, 10, or 40 trials of a counterclockwise 30° cursor rotation during reaching movements on one day and were retested the following day to assay for savings. Groups that experienced five trials or more of adaptation on day 1 showed clear savings on day 2. Subjects in all groups learned significantly more from the first rotation trial on day 2 than on day 1, but this learning rate advantage was maintained only in groups that had reached asymptote during the initial exposure. Additional experiments revealed that savings occurred when the magnitude, but not the direction, of the rotation differed across exposures, and when a 5-min break, rather than an overnight one, separated the first and second exposure. The overall pattern of savings we observe across conditions can be explained as rapid retrieval of the state of learning attained during the first exposure rather than as modulation of sensitivity to error. We conclude that a long-term memory for compensating for a perturbation can be rapidly acquired and rapidly retrieved.

Implicit guidance to stable performance in a rhythmic perceptual-motor skill

Feedback about error or reward is regarded essential for aiding learners to acquire a perceptual-motor skill. Yet, when a task has redundancy and the mapping between execution and performance outcome is unknown, simple error feedback does not suffice in guiding the learner toward the optimal solutions. The present study developed and tested a new means of implicitly guiding learners to acquire a perceptual-motor skill, rhythmically bouncing a ball on a racket. Due to its rhythmic nature, this task affords dynamically stable solutions that are robust to small errors and noise, a strategy that is independent from actively correcting error. Based on the task model implemented in a virtual environment, a time-shift manipulation was designed to shift the range of ball-racket contacts that achieved dynamically stable solutions. In two experiments, subjects practiced with this manipulation that guided them to impact the ball with more negative racket accelerations, the indicator for the strategy with dynamic stability. Subjects who practiced under normal conditions took longer time to acquire this strategy, although error measures were identical between the control and experimental groups. Unlike in many other haptic guidance or adaptation studies, the experimental groups not only learned, but also maintained the stable solution after the manipulation was removed. These results are a first demonstration that more subtle ways to guide the learner to better performance are needed especially in tasks with redundancy, where error feedback may not be sufficient.

Brain structural substrates of reward dependence during behavioral performance

Interindividual differences in the effects of reward on performance are prevalent and poorly understood, with some individuals being more dependent than others on the rewarding outcomes of their actions. The origin of this variability in reward dependence is unknown. Here, we tested the relationship between reward dependence and brain structure in healthy humans. Subjects trained on a visuomotor skill-acquisition task and received performance feedback in the presence or absence of reward. Reward dependence was defined as the statistical trial-by-trial relation between reward and subsequent performance. We report a significant relationship between reward dependence and the lateral prefrontal cortex, where regional gray-matter volume predicted reward dependence but not feedback alone. Multivoxel pattern analysis confirmed the anatomical specificity of this relationship. These results identified a likely anatomical marker for the prospective influence of reward on performance, which may be of relevance in neurorehabilitative settings.

Reward feedback accelerates motor learning

Recent findings have demonstrated that reward feedback alone can drive motor learning. However, it is not yet clear whether reward feedback alone can lead to learning when a perturbation is introduced abruptly, or how a reward gradient can modulate learning. In this study, we provide reward feedback that decays continuously with increasing error. We asked whether it is possible to learn an abrupt visuomotor rotation by reward alone, and if the learning process could be modulated by combining reward and sensory feedback and/or by using different reward landscapes. We designed a novel visuomotor learning protocol during which subjects experienced an abruptly introduced rotational perturbation. Subjects received either visual feedback or reward feedback, or a combination of the two. Two different reward landscapes, where the reward decayed either linearly or cubically with distance from the target, were tested. Results demonstrate that it is possible to learn from reward feedback alone and that the combination of reward and sensory feedback accelerates learning. An analysis of the underlying mechanisms reveals that although reward feedback alone does not allow for sensorimotor remapping, it can nonetheless lead to broad generalization, highlighting a dissociation between remapping and generalization. Also, the combination of reward and sensory feedback accelerates learning without compromising sensorimotor remapping. These findings suggest that the use of reward feedback is a promising approach to either supplement or substitute sensory feedback in the development of improved neurorehabilitation techniques. More generally, they point to an important role played by reward in the motor learning process.