Dissociating error-based and reinforcement-based loss functions during sensorimotor learning

Exploration-based learning of a stabilizing controller predicts locomotor adaptation

Humans adapt their locomotion seamlessly in response to changes in the body or the environment. It is unclear how such adaptation improves performance measures like energy consumption or symmetry while avoiding falling. Here, we model locomotor adaptation as interactions between a stabilizing controller that reacts quickly to perturbations and a reinforcement learner that gradually improves the controller's performance through local exploration and memory. This model predicts time-varying adaptation in many settings: walking on a split-belt treadmill (i.e. with both feet at different speeds), with asymmetric leg weights, or using exoskeletons - capturing learning and generalization phenomena in ten prior experiments and two model-guided experiments conducted here. The performance measure of energy minimization with a minor cost for asymmetry captures a broad range of phenomena and can act alongside other mechanisms such as reducing sensory prediction error. Such a model-based understanding of adaptation can guide rehabilitation and wearable robot control.

Touchscreen response precision is sensitive to the explore/exploit tradeoff

The explore/exploit tradeoff is a fundamental property of choice selection during reward-guided decision making. In perceptual decision making, higher certainty decisions are more motorically precise, even when the decision does not require motor accuracy. However, while we can parametrically control uncertainty in perceptual tasks, we do not know what variables - if any - shape motor precision and reflect subjective certainty during reward-guided decision making. Touchscreens are increasingly used across species to measure choice, but provide no tactile feedback on whether an action is precise or not, and therefore provide a valuable opportunity to determine whether actions differ in precision due to explore/exploit state, reward, or individual variables. We find all three of these factors exert independent drives towards increased precision. During exploit states, successive touches to the same choice are closer together than those made in an explore state, consistent with exploit states reflecting higher certainty and/or motor stereotypy in responding. However, exploit decisions might be expected to be rewarded more frequently than explore decisions. We find that exploit choice precision is increased independently of a separate increase in precision due to immediate past reward, suggesting multiple mechanisms regulating choice precision. Finally, we see evidence that male mice in general are less precise in their interactions with the touchscreen than females, even when exploiting a choice. These results suggest that as exploit behavior emerges in reward-guided decision making, individuals become more motorically precise reflecting increased certainty, even when decision choice does not require additional motor accuracy, but this is influenced by individual differences and prior reward. These data uncover the hidden potential for touchscreen tasks in any species to uncover the latent neural states that unite cognition and movement.

Roles and interplay of reinforcement-based and error-based processes during reaching and gait in neurotypical adults and individuals with Parkinson's disease

From a game of darts to neurorehabilitation, the ability to explore and fine tune our movements is critical for success. Past work has shown that exploratory motor behaviour in response to reinforcement (reward) feedback is closely linked with the basal ganglia, while movement corrections in response to error feedback is commonly attributed to the cerebellum. While our past work has shown these processes are dissociable during adaptation, it is unknown how they uniquely impact exploratory behaviour. Moreover, converging neuroanatomical evidence shows direct and indirect connections between the basal ganglia and cerebellum, suggesting that there is an interaction between reinforcement-based and error-based neural processes. Here we examine the unique roles and interaction between reinforcement-based and error-based processes on sensorimotor exploration in a neurotypical population. We also recruited individuals with Parkinson's disease to gain mechanistic insight into the role of the basal ganglia and associated reinforcement pathways in sensorimotor exploration. Across three reaching experiments, participants were given either reinforcement feedback, error feedback, or simultaneously both reinforcement & error feedback during a sensorimotor task that encouraged exploration. Our reaching results, a re-analysis of a previous gait experiment, and our model suggests that in isolation, reinforcement-based and error-based processes respectively boost and suppress exploration. When acting in concert, we found that reinforcement-based and error-based processes interact by mutually opposing one another. Finally, we found that those with Parkinson's disease had decreased exploration when receiving reinforcement feedback, supporting the notion that compromised reinforcement-based processes reduces the ability to explore new motor actions. Understanding the unique and interacting roles of reinforcement-based and error-based processes may help to inform neurorehabilitation paradigms where it is important to discover new and successful motor actions.

Human motor learning dynamics in high-dimensional tasks

Conventional approaches to enhance movement coordination, such as providing instructions and visual feedback, are often inadequate in complex motor tasks with multiple degrees of freedom (DoFs). To effectively address coordination deficits in such complex motor systems, it becomes imperative to develop interventions grounded in a model of human motor learning; however, modeling such learning processes is challenging due to the large DoFs. In this paper, we present a computational motor learning model that leverages the concept of motor synergies to extract low-dimensional learning representations in the high-dimensional motor space and the internal model theory of motor control to capture both fast and slow motor learning processes. We establish the model's convergence properties and validate it using data from a target capture game played by human participants. We study the influence of model parameters on several motor learning trade-offs such as speed-accuracy, exploration-exploitation, satisficing, and flexibility-performance, and show that the human motor learning system tunes these parameters to optimize learning and various output performance metrics.

Non-invasive stimulation of the human striatum disrupts reinforcement learning of motor skills

Reinforcement feedback can improve motor learning, but the underlying brain mechanisms remain underexplored. In particular, the causal contribution of specific patterns of oscillatory activity within the human striatum is unknown. To address this question, we exploited a recently developed non-invasive deep brain stimulation technique called transcranial temporal interference stimulation (tTIS) during reinforcement motor learning with concurrent neuroimaging, in a randomized, sham-controlled, double-blind study. Striatal tTIS applied at 80 Hz, but not at 20 Hz, abolished the benefits of reinforcement on motor learning. This effect was related to a selective modulation of neural activity within the striatum. Moreover, 80 Hz, but not 20 Hz, tTIS increased the neuromodulatory influence of the striatum on frontal areas involved in reinforcement motor learning. These results show that tTIS can non-invasively and selectively modulate a striatal mechanism involved in reinforcement learning, expanding our tools for the study of causal relationships between deep brain structures and human behaviour.

Reinforcement Motor Learning After Cerebellar Damage Is Related to State Estimation

Recent work showed that individuals with cerebellar degeneration could leverage intact reinforcement learning (RL) to alter their movement. However, there was marked inter-individual variability in learning, and the factors underlying it were unclear. Cerebellum-dependent sensory prediction may contribute to RL in motor contexts by enhancing body state estimates, which are necessary to solve the credit-assignment problem. The objective of this study was to test the relationship between the predictive component of state estimation and RL in individuals with cerebellar degeneration. Individuals with cerebellar degeneration and neurotypical control participants completed two tasks: an RL task that required them to alter the angle of reaching movements and a state estimation task that tested the somatosensory perception of active and passive movement. The state estimation task permitted the calculation of the active benefit shown by each participant, which is thought to reflect the cerebellum-dependent predictive component of state estimation. We found that the cerebellar and control groups showed similar magnitudes of learning with reinforcement and active benefit on average, but there was substantial variability across individuals. Using multiple regression, we assessed potential predictors of RL. Our analysis included active benefit, somatosensory acuity, clinical ataxia severity, movement variability, movement speed, and age. We found a significant relationship in which greater active benefit predicted better learning with reinforcement in the cerebellar, but not the control group. No other variables showed significant relationships with learning. Overall, our results support the hypothesis that the integrity of sensory prediction is a strong predictor of RL after cerebellar damage.

Oculomotor behavior can be adjusted on the basis of artificial feedback signals indicating externally caused errors

Whether a saccade is accurate and has reached the target cannot be evaluated during its execution, but relies on post-saccadic feedback. If the eye has missed the target object, a secondary corrective saccade has to be made to align the fovea with the target. If a systematic post-saccadic error occurs, adaptive changes to the oculomotor behavior are made, such as shortening or lengthening the saccade amplitude. Systematic post-saccadic errors are typically attributed internally to erroneous motor commands. The corresponding adaptive changes to the motor command reduce the error and the need for secondary corrective saccades, and, in doing so, restore accuracy and efficiency. However, adaptive changes to the oculomotor behavior also occur if a change in saccade amplitude is beneficial for task performance, or if it is rewarded. Oculomotor learning thus is more complex than reducing a post-saccadic position error. In the current study, we used a novel oculomotor learning paradigm and investigated whether human participants are able to adapt their oculomotor behavior to improve task performance even when they attribute the error externally. The task was to indicate the intended target object among several objects to a simulated human-machine interface by making eye movements. The participants were informed that the system itself could make errors. The decoding process depended on a distorted landing point of the saccade, resulting in decoding errors. Two different types of visual feedback were added to the post-saccadic scene and we compared how participants used the different feedback types to adjust their oculomotor behavior to avoid errors. We found that task performance improved over time, regardless of the type of feedback. Thus, error feedback from the simulated human-machine interface was used for post-saccadic error evaluation. This indicates that 1) artificial visual feedback signals and 2) externally caused errors might drive adaptive changes to oculomotor behavior.

Punishment Leads to Greater Sensorimotor Learning But Less Movement Variability Compared to Reward

When a musician practices a new song, hitting a correct note sounds pleasant while striking an incorrect note sounds unpleasant. Such reward and punishment feedback has been shown to differentially influence the ability to learn a new motor skill. Recent work has suggested that punishment leads to greater movement variability, which causes greater exploration and faster learning. To further test this idea, we collected 102 participants over two experiments. Unlike previous work, in Experiment 1 we found that punishment did not lead to faster learning compared to reward (n = 68), but did lead to a greater extent of learning. Surprisingly, we also found evidence to suggest that punishment led to less movement variability, which was related to the extent of learning. We then designed a second experiment that did not involve adaptation, allowing us to further isolate the influence of punishment feedback on movement variability. In Experiment 2, we again found that punishment led to significantly less movement variability compared to reward (n = 34). Collectively our results suggest that punishment feedback leads to less movement variability. Future work should investigate whether punishment feedback leads to a greater knowledge of movement variability and or increases the sensitivity of updating motor actions.

Reinforcement Learning during Locomotion

When learning a new motor skill, people often must use trial and error to discover which movement is best. In the reinforcement learning framework, this concept is known as exploration and has been linked to increased movement variability in motor tasks. For locomotor tasks, however, increased variability decreases upright stability. As such, exploration during gait may jeopardize balance and safety, making reinforcement learning less effective. Therefore, we set out to determine if humans could acquire and retain a novel locomotor pattern using reinforcement learning alone. Young healthy male and female participants walked on a treadmill and were provided with binary reward feedback (indicated by a green checkmark on the screen) that was tied to a fixed monetary bonus, to learn a novel stepping pattern. We also recruited a comparison group who walked with the same novel stepping pattern but did so by correcting for target error, induced by providing real-time veridical visual feedback of steps and a target. In two experiments, we compared learning, motor variability, and two forms of motor memories between the groups. We found that individuals in the binary reward group did, in fact, acquire the new walking pattern by exploring (increasing motor variability). Additionally, while reinforcement learning did not increase implicit motor memories, it resulted in more accurate explicit motor memories compared with the target error group. Overall, these results demonstrate that humans can acquire new walking patterns with reinforcement learning and retain much of the learning over 24 h.

NSF DARE-transforming modeling in neurorehabilitation: a patient-in-the-loop framework

In 2023, the National Science Foundation (NSF) and the National Institute of Health (NIH) brought together engineers, scientists, and clinicians by sponsoring a conference on computational modelling in neurorehabiilitation. To facilitate multidisciplinary collaborations and improve patient care, in this perspective piece we identify where and how computational modelling can support neurorehabilitation. To address the where, we developed a patient-in-the-loop framework that uses multiple and/or continual measurements to update diagnostic and treatment model parameters, treatment type, and treatment prescription, with the goal of maximizing clinically-relevant functional outcomes. This patient-in-the-loop framework has several key features: (i) it includes diagnostic and treatment models, (ii) it is clinically-grounded with the International Classification of Functioning, Disability and Health (ICF) and patient involvement, (iii) it uses multiple or continual data measurements over time, and (iv) it is applicable to a range of neurological and neurodevelopmental conditions. To address the how, we identify state-of-the-art and highlight promising avenues of future research across the realms of sensorimotor adaptation, neuroplasticity, musculoskeletal, and sensory & pain computational modelling. We also discuss both the importance of and how to perform model validation, as well as challenges to overcome when implementing computational models within a clinical setting. The patient-in-the-loop approach offers a unifying framework to guide multidisciplinary collaboration between computational and clinical stakeholders in the field of neurorehabilitation.

Dorsomedial frontal cortex damage impairs error-based, but not reinforcement-based motor learning in humans

We adapt our movements to new and changing environments through multiple processes. Sensory error-based learning counteracts environmental perturbations that affect the sensory consequences of movements. Sensory errors also cause the upregulation of reflexes and muscle co-contraction. Reinforcement-based learning enhances the selection of movements that produce rewarding outcomes. Although some findings have identified dissociable neural substrates of sensory error- and reinforcement-based learning, correlative methods have implicated dorsomedial frontal cortex in both. Here, we tested the causal contributions of dorsomedial frontal to adaptive motor control, studying people with chronic damage to this region. Seven human participants with focal brain lesions affecting the dorsomedial frontal and 20 controls performed a battery of arm movement tasks. Three experiments tested: (i) the upregulation of visuomotor reflexes and muscle co-contraction in response to unpredictable mechanical perturbations, (ii) sensory error-based learning in which participants learned to compensate predictively for mechanical force-field perturbations, and (iii) reinforcement-based motor learning based on binary feedback in the absence of sensory error feedback. Participants with dorsomedial frontal damage were impaired in the early stages of force field adaptation, but performed similarly to controls in all other measures. These results provide evidence for a specific and selective causal role for the dorsomedial frontal in sensory error-based learning.

Limitations in utilization and prioritization of standardized somatosensory assessments after stroke: A cross-sectional survey of neurorehabilitation clinicians

BACKGROUND AND PURPOSE

Somatosensory impairments are common after stroke, but receive limited evaluation and intervention during neurorehabilitation, despite negatively impacting functional movement and recovery.

OBJECTIVES

Our objective was to understand the scope of somatosensory assessments used by clinicians in stroke rehabilitation, and barriers to increasing use in clinical practice.

METHODS

An electronic survey was distributed to clinicians (physical therapists, occupational therapists, physicians, and nurses) who assessed at least one individual with stroke in the past 6 months. The survey included questions on evaluation procedures, type, and use of somatosensory assessments, as well as barriers and facilitators in clinical practice.

RESULTS

Clinicians (N = 431) indicated greater familiarity with non-standardized assessments, and greater utilization compared to standardized assessments (p < 0.0001). Components of tactile sensation were the most commonly assessed modality of somatosensation (25%), while proprioception was rarely assessed (1%). Overall, assessments of motor function were prioritized over assessments of somatosensory function (p < 0.0001).

DISCUSSION

Respondents reported assessing somatosensation less frequently than motor function and demonstrated a reliance on rapid and coarse non-standardized assessments that ineffectively capture multi-modal somatosensory impairments, particularly for proprioceptive deficits common post-stroke. In general, clinicians were not familiar with standardized somatosensory assessments, and this knowledge gap likely contributes to lack of translation of these assessments into practice.

CONCLUSIONS

Clinicians utilize somatosensory assessments that inadequately capture the multi-modal nature of somatosensory impairments in stroke survivors. Addressing barriers to clinical translation has the potential to increase utilization of standardized assessments to improve the characterization of somatosensory deficits that inform clinical decision-making toward enhancing stroke rehabilitation outcomes.

Behavioral Motor Performance

The human sensorimotor control system has exceptional abilities to perform skillful actions. We easily switch between strenuous tasks that involve brute force, such as lifting a heavy sewing machine, and delicate movements such as threading a needle in the same machine. Using a structure with different control architectures, the motor system is capable of updating its ability to perform through our daily interaction with the fluctuating environment. However, there are issues that make this a difficult computational problem for the brain to solve. The brain needs to control a nonlinear, nonstationary neuromuscular system, with redundant and occasionally undesired degrees of freedom, in an uncertain environment using a body in which information transmission is subject to delays and noise. To gain insight into the mechanisms of motor control, here we survey movement laws and invariances that shape our everyday motion. We then examine the major solutions to each of these problems in the three parts of the sensorimotor control system, sensing, planning, and acting. We focus on how the sensory system, the control architectures, and the structure and operation of the muscles serve as complementary mechanisms to overcome deviations and disturbances to motor behavior and give rise to skillful motor performance. We conclude with possible future research directions based on suggested links between the operation of the sensorimotor system across the movement stages. © 2024 American Physiological Society. Compr Physiol 14:5179-5224, 2024.

Differential effects of reward and punishment on reinforcement-based motor learning and generalization

Reward and punishment have long been recognized as potent modulators of human behavior. Although reinforcement learning is a significant motor learning process, the exact mechanisms underlying how the brain learns movements through reward and punishment are not yet fully understood. Beyond the memory of specific examples, investigating the ability to generalize to new situations offers a better understanding of motor learning. This study hypothesizes that reward and punishment engage qualitatively different motivational systems with different neurochemical and neuroanatomical substrates, which would have differential effects on reinforcement-based motor learning and generalization. To test this hypothesis, two groups of participants learn a motor task in one direction and then relearn the same task in a new direction, receiving only performance-based reward or punishment score feedback. Our findings support our hypothesis, showing that reward led to slower learning but promoted generalization. On the other hand, punishment led to faster learning but impaired generalization. These behavioral differences may be due to different tendencies of movement variability in each group. The punishment group tended to explore more actively than the reward group during the initial learning phase, possibly due to loss aversion. In contrast, the reward group tended to explore more actively than the initial learning phase during the generalization test phase, seemingly recalling the strategy that led to the reward. These results suggest that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization, with important implications for practical applications such as sports training and motor rehabilitation.NEW & NOTEWORTHY Although reinforcement learning is a significant motor learning process, the mechanisms underlying how the brain learns movements through reward and punishment are not fully understood. We modified a well-established motor adaptation task and used savings (faster relearning) to measure generalization. We found reward led to slower learning but promoted generalization, whereas punishment led to faster learning but impaired generalization, suggesting that reward and punishment may engage different neural mechanisms during reinforcement-based motor learning and generalization.

Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds

From a baby's babbling to a songbird practising a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species' neural firing, gait patterns and reaching behaviour. The past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here, we test a fundamentally different idea-that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration to maximize success. Across three human reaching experiments, we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore muscle, joint and task redundant solution manifolds is beneficial while acting in uncertain environments, during motor development or when recovering from a neurological disorder to discover and learn new motor actions.

Exploring the role of task success in implicit motor adaptation

Although implicit motor adaptation is driven by sensory-prediction errors (SPEs), recent work has shown that task success modulates this process. Task success has typically been defined as hitting a target, which signifies the goal of the movement. Visuomotor adaptation tasks are uniquely situated to experimentally manipulate task success independently from SPE by changing the target size or the location of the target. These two, distinct manipulations may influence implicit motor adaptation in different ways, so, over four experiments, we sought to probe the efficacy of each manipulation. We found that changes in target size, which caused the target to fully envelop the cursor, only affected implicit adaptation for a narrow range of SPE sizes, while jumping the target to overlap with the cursor more reliably and robustly affected implicit adaptation. Taken together, our data indicate that, while task success exerts a small effect on implicit adaptation, these effects are susceptible to methodological variations. Future investigations of the effect of task success on implicit adaptation could benefit from employing target jump manipulations instead of target size manipulations.NEW & NOTEWORTHY Recent work has suggested that task success, namely, hitting a target, influences implicit motor adaptation. Here, we observed that implicit adaptation is modulated by target jump manipulations, where the target abruptly "jumps" to meet the cursor; however, implicit adaptation was only weakly modulated by target size manipulations, where a static target either envelops or excludes the cursor. We discuss how these manipulations may exert their effects through different mechanisms.

Exploring the role of task success in implicit motor adaptation

We learn to improve our motor skills using different forms of feedback: sensory-prediction error, task success, and reward/punishment. While implicit motor adaptation is driven by sensory-prediction errors, recent work has shown that task success modulates this process. Task success is often confounded with reward, so we sought to determine if the effects of these two signals on adaptation can be dissociated. To address this question, we conducted five experiments that isolated implicit learning using error-clamp visuomotor reach adaptation paradigms. Task success was manipulated by changing the size and position of the target relative to the cursor providing visual feedback, and reward expectation was established using monetary cues and auditory feedback. We found that neither monetary cues nor auditory feedback affected implicit adaptation, suggesting that task success influences implicit adaptation via mechanisms distinct from conventional reward-related processes. Additionally, we found that changes in target size, which caused the target to either exclude or fully envelop the cursor, only affected implicit adaptation for a narrow range of error sizes, while jumping the target to overlap with the cursor more reliably and robustly affected implicit adaptation. Taken together, our data indicate that, while task success exerts a small effect on implicit adaptation, these effects are susceptible to methodological variations and unlikely to be mediated by reward.

The sensorimotor system modulates muscular co-contraction relative to visuomotor feedback responses to regulate movement variability

The naturally occurring variability in our movements often poses a significant challenge when attempting to produce precise and accurate actions, which is readily evident when playing a game of darts. Two differing, yet potentially complementary, control strategies that the sensorimotor system may use to regulate movement variability are impedance control and feedback control. Greater muscular co-contraction leads to greater impedance that acts to stabilize the hand, while visuomotor feedback responses can be used to rapidly correct for unexpected deviations when reaching toward a target. Here, we examined the independent roles and potential interplay of impedance control and visuomotor feedback control when regulating movement variability. Participants were instructed to perform a precise reaching task by moving a cursor through a narrow visual channel. We manipulated cursor feedback by visually amplifying movement variability and/or delaying the visual feedback of the cursor. We found that participants decreased movement variability by increasing muscular co-contraction, aligned with an impedance control strategy. Participants displayed visuomotor feedback responses during the task but, unexpectedly, there was no modulation between conditions. However, we did find a relationship between muscular co-contraction and visuomotor feedback responses, suggesting that participants modulated impedance control relative to feedback control. Taken together, our results highlight that the sensorimotor system modulates muscular co-contraction, relative to visuomotor feedback responses, to regulate movement variability and produce accurate actions.NEW & NOTEWORTHY The sensorimotor system has the constant challenge of dealing with the naturally occurring variability in our movements. Here, we investigated the potential roles of muscular co-contraction and visuomotor feedback responses to regulate movement variability. When we visually amplified movements, we found that the sensorimotor system primarily uses muscular co-contraction to regulate movement variability. Interestingly, we found that muscular co-contraction was modulated relative to inherent visuomotor feedback responses, suggesting an interplay between impedance and feedback control.

Error-based and reinforcement learning in basketball free throw shooting

This study investigates the effects of error-based and reinforcement training on the acquisition and long-term retention of free throw accuracy in basketball. Sixty participants were divided into four groups (n = 15 per group): (i) the error-based group (sensory feedback), (ii) the reinforcement group (binary feedback including success or failure), (iii) the mixed group (sensory feedback followed by binary feedback), and (iv) the control group (without training). Free throws success was recorded before training (PreT), immediately after (Postd0), one day later (Postd1), and seven days later (Postd7). The error-based group, but not the reinforcement group, showed a significant immediate improvement in free throw accuracy (PreT vs Postd0). Interestingly, over time (Postd0 vs Postd1 vs Postd7), the reinforcement group significantly improved its accuracy, while the error-based group decreased it, returning to the PreT level (PreT vs Post7). The mixed group showed the advantage of both training methods, i.e., fast acquisition and retention on a long-term scale. Error-based learning leads to better acquisition, while reinforcement learning leads to better retention. Therefore, the combination of both types of learning is more efficient for both acquisition and retention processes. These findings provide new insight into the acquisition and retention of a fundamental basketball skill in free throw shooting.

Failure induces task-irrelevant exploration during a stencil task

During reward-based motor tasks, performance failure leads to an increase in movement variability along task-relevant dimensions. These increases in movement variability are indicative of exploratory behaviour in search of a better, more successful motor action. It is unclear whether failure also induces exploration along task-irrelevant dimensions that do not influence performance. In this study, we ask whether participants would explore the task-irrelevant dimension while they performed a stencil task. With a stylus, participants applied downward, normal force that influenced whether they received reward (task-relevant) as they simultaneously made erasing-like movement patterns along the tablet that did not influence performance (task-irrelevant). In this task, the movement pattern was analyzed as the distribution of movement directions within a movement. The results showed significant exploration of task-relevant force and task-irrelevant movement patterns. We conclude that failure can induce additional movement variability along a task-irrelevant dimension.

Shaping Exploration: How Does the Constraint-Induced Movement Therapy Helps Patients Finding a New Movement Solution

Despite the relative success of constraint-induced movement therapy in the recovery of injury-/trauma-related populations, the mechanisms by which it promotes its results are still unknown. From a dynamical systems approach, we investigated whether the induced exploratory patterns within and between trials during an exercise in Shaping (the therapy's practice) could shed light on this process. We analyzed data from four chronic spinal-cord injury patients during a task of placing and removing their feet from a step. We assessed the within and between trial dynamics through recurrent quantification analyses and task-space analyses, respectively. From our results, individuals found movement patterns directed to modulate foot height (to accomplish the task). Additionally, when the task was manipulated (increasing step height), individuals increased coupling and coupling variability in the ankle, hip, and knee over trials. This pattern of findings is in consonance with the idea of Shaping inducing exploration of different movements. Such exploration might be an important factor affording the positive changes observed in the literature.

Exploration of sensory-motor tradeoff behavior in Parkinson's disease

While slowness of movement is an obligatory characteristic of Parkinson's disease (PD), there are conditions in which patients move uncharacteristically fast, attributed to deficient motor inhibition. Here we investigate deficient inhibition in an optimal sensory-motor integration framework, using a game in which subjects used a paddle to catch a virtual ball. Display of the ball was extinguished as soon as the catching movement started, segregating the task into a sensing and acting phase. We analyzed the behavior of 9 PD patients (ON medication) and 10 age-matched controls (HC). The switching times (between sensing and acting phase) were compared to the predicted optimal switching time, based on the individual estimates of sensory and motor uncertainties. The comparison showed that deviation from predicted optimal switching times were similar between groups. However, PD patients showed a weaker correlation between variability in switching time and sensory-motor uncertainty, indicating a reduced propensity to generate exploratory behavior for optimizing goal-directed movements. Analysis of the movement kinematics revealed that PD patients, compared to controls, used a lower peak velocity of the paddle and intercepted the ball with greater velocity. Adjusting the trial duration to the time for the paddle to stop moving, we found that PD patients spent a smaller proportion of the trial duration for observing the ball. Altogether, the results do not show the premature movement initiation and truncated sensory processing that we predicted to ensue from deficient inhibition in PD.

Exercising choice over feedback schedules during practice is not advantageous for motor learning

The idea that there is a self-controlled learning advantage, where individuals demonstrate improved motor learning after exercising choice over an aspect of practice compared to no-choice groups, has different causal explanations according to the OPTIMAL theory or an information-processing perspective. Within OPTIMAL theory, giving learners choice is considered an autonomy-supportive manipulation that enhances expectations for success and intrinsic motivation. In the information-processing view, choice allows learners to engage in performance-dependent strategies that reduce uncertainty about task outcomes. To disentangle these potential explanations, we provided participants in choice and yoked groups with error or graded feedback (Experiment 1) and binary feedback (Experiment 2) while learning a novel motor task with spatial and timing goals. Across both experiments (N = 228 participants), we did not find any evidence to support a self-controlled learning advantage. Exercising choice during practice did not increase perceptions of autonomy, competence, or intrinsic motivation, nor did it lead to more accurate error estimation skills. Both error and graded feedback facilitated skill acquisition and learning, whereas no improvements from pre-test performance were found with binary feedback. Finally, the impact of graded and binary feedback on perceived competence highlights a potential dissociation of motivational and informational roles of feedback. Although our results regarding self-controlled practice conditions are difficult to reconcile with either the OPTIMAL theory or the information-processing perspective, they are consistent with a growing body of evidence that strongly suggests self-controlled conditions are not an effective approach to enhance motor performance and learning.

Human Variation in Error-Based and Reinforcement Motor Learning Is Associated With Entorhinal Volume

Error-based and reward-based processes are critical for motor learning and are thought to be mediated via distinct neural pathways. However, recent behavioral work in humans suggests that both learning processes can be bolstered by the use of cognitive strategies, which may mediate individual differences in motor learning ability. It has been speculated that medial temporal lobe regions, which have been shown to support motor sequence learning, also support the use of cognitive strategies in error-based and reinforcement motor learning. However, direct evidence in support of this idea remains sparse. Here we first show that better overall learning during error-based visuomotor adaptation is associated with better overall learning during the reward-based shaping of reaching movements. Given the cognitive contribution to learning in both of these tasks, these results support the notion that strategic processes, associated with better performance, drive intersubject variation in both error-based and reinforcement motor learning. Furthermore, we show that entorhinal cortex volume is larger in better learning individuals-characterized across both motor learning tasks-compared with their poorer learning counterparts. These results suggest that individual differences in learning performance during error and reinforcement learning are related to neuroanatomical differences in entorhinal cortex.

Reach adaption to a visuomotor gain with terminal error feedback involves reinforcement learning

Motor adaptation can be achieved through error-based learning, driven by sensory prediction errors, or reinforcement learning, driven by reward prediction errors. Recent work on visuomotor adaptation has shown that reinforcement learning leads to more persistent adaptation when visual feedback is removed, compared to error-based learning in which continuous visual feedback of the movement is provided. However, there is evidence that error-based learning with terminal visual feedback of the movement (provided at the end of movement) may be driven by both sensory and reward prediction errors. Here we examined the influence of feedback on learning using a visuomotor adaptation task in which participants moved a cursor to a single target while the gain between hand and cursor movement displacement was gradually altered. Different groups received either continuous error feedback (EC), terminal error feedback (ET), or binary reinforcement feedback (success/fail) at the end of the movement (R). Following adaptation we tested generalization to targets located in different directions and found that generalization in the ET group was intermediate between the EC and R groups. We then examined the persistence of adaptation in the EC and ET groups when the cursor was extinguished and only binary reward feedback was provided. Whereas performance was maintained in the ET group, it quickly deteriorated in the EC group. These results suggest that terminal error feedback leads to a more robust form of learning than continuous error feedback. In addition our findings are consistent with the view that error-based learning with terminal feedback involves both error-based and reinforcement learning.

Humans utilize sensory evidence of others' intended action to make online decisions

We often acquire sensory information from another person's actions to make decisions on how to move, such as when walking through a crowded hallway. Past interactive decision-making research has focused on cognitive tasks that did not allow for sensory information exchange between humans prior to a decision. Here, we test the idea that humans accumulate sensory evidence of another person's intended action to decide their own movement. In a competitive sensorimotor task, we show that humans exploit time to accumulate sensory evidence of another's intended action and utilize this information to decide how to move. We captured this continuous interactive decision-making behaviour with a drift-diffusion model. Surprisingly, aligned with a 'paralysis-by-analysis' phenomenon, we found that humans often waited too long to accumulate sensory evidence and failed to make a decision. Understanding how humans engage in interactive and online decision-making has broad implications that spans sociology, athletics, interactive technology, and economics.

Using Artificial Intelligence for Assistance Systems to Bring Motor Learning Principles into Real World Motor Tasks

Humans learn movements naturally, but it takes a lot of time and training to achieve expert performance in motor skills. In this review, we show how modern technologies can support people in learning new motor skills. First, we introduce important concepts in motor control, motor learning and motor skill learning. We also give an overview about the rapid expansion of machine learning algorithms and sensor technologies for human motion analysis. The integration between motor learning principles, machine learning algorithms and recent sensor technologies has the potential to develop AI-guided assistance systems for motor skill training. We give our perspective on this integration of different fields to transition from motor learning research in laboratory settings to real world environments and real world motor tasks and propose a stepwise approach to facilitate this transition.

Interactions between sensory prediction error and task error during implicit motor learning

Implicit motor recalibration allows us to flexibly move in novel and changing environments. Conventionally, implicit recalibration is thought to be driven by errors in predicting the sensory outcome of movement (i.e., sensory prediction errors). However, recent studies have shown that implicit recalibration is also influenced by errors in achieving the movement goal (i.e., task errors). Exactly how sensory prediction errors and task errors interact to drive implicit recalibration and, in particular, whether task errors alone might be sufficient to drive implicit recalibration remain unknown. To test this, we induced task errors in the absence of sensory prediction errors by displacing the target mid-movement. We found that task errors alone failed to induce implicit recalibration. In additional experiments, we simultaneously varied the size of sensory prediction errors and task errors. We found that implicit recalibration driven by sensory prediction errors could be continuously modulated by task errors, revealing an unappreciated dependency between these two sources of error. Moreover, implicit recalibration was attenuated when the target was simply flickered in its original location, even though this manipulation did not affect task error - an effect likely attributed to attention being directed away from the feedback cursor. Taken as a whole, the results were accounted for by a computational model in which sensory prediction errors and task errors, modulated by attention, interact to determine the extent of implicit recalibration.

Mechanisms of Human Motor Learning Do Not Function Independently

Human motor learning is governed by a suite of interacting mechanisms each one of which modifies behavior in distinct ways and rely on different neural circuits. In recent years, much attention has been given to one type of motor learning, called motor adaptation. Here, the field has generally focused on the interactions of three mechanisms: sensory prediction error SPE-driven, explicit (strategy-based), and reinforcement learning. Studies of these mechanisms have largely treated them as modular, aiming to model how the outputs of each are combined in the production of overt behavior. However, when examined closely the results of some studies also suggest the existence of additional interactions between the sub-components of each learning mechanism. In this perspective, we propose that these sub-component interactions represent a critical means through which different motor learning mechanisms are combined to produce movement; understanding such interactions is critical to advancing our knowledge of how humans learn new behaviors. We review current literature studying interactions between SPE-driven, explicit, and reinforcement mechanisms of motor learning. We then present evidence of sub-component interactions between SPE-driven and reinforcement learning as well as between SPE-driven and explicit learning from studies of people with cerebellar degeneration. Finally, we discuss the implications of interactions between learning mechanism sub-components for future research in human motor learning.

Reward boosts reinforcement-based motor learning

Besides relying heavily on sensory and reinforcement feedback, motor skill learning may also depend on the level of motivation experienced during training. Yet, how motivation by reward modulates motor learning remains unclear. In 90 healthy subjects, we investigated the net effect of motivation by reward on motor learning while controlling for the sensory and reinforcement feedback received by the participants. Reward improved motor skill learning beyond performance-based reinforcement feedback. Importantly, the beneficial effect of reward involved a specific potentiation of reinforcement-related adjustments in motor commands, which concerned primarily the most relevant motor component for task success and persisted on the following day in the absence of reward. We propose that the long-lasting effects of motivation on motor learning may entail a form of associative learning resulting from the repetitive pairing of the reinforcement feedback and reward during training, a mechanism that may be exploited in future rehabilitation protocols.

De novo learning versus adaptation of continuous control in a manual tracking task

How do people learn to perform tasks that require continuous adjustments of motor output, like riding a bicycle? People rely heavily on cognitive strategies when learning discrete movement tasks, but such time-consuming strategies are infeasible in continuous control tasks that demand rapid responses to ongoing sensory feedback. To understand how people can learn to perform such tasks without the benefit of cognitive strategies, we imposed a rotation/mirror reversal of visual feedback while participants performed a continuous tracking task. We analyzed behavior using a system identification approach, which revealed two qualitatively different components of learning: adaptation of a baseline controller and formation of a new, task-specific continuous controller. These components exhibited different signatures in the frequency domain and were differentially engaged under the rotation/mirror reversal. Our results demonstrate that people can rapidly build a new continuous controller de novo and can simultaneously deploy this process with adaptation of an existing controller.

Null effects of levodopa on reward- and error-based motor adaptation, savings, and anterograde interference

Dopamine signaling is thought to mediate reward-based learning. We tested for a role of dopamine in motor adaptation by administering the dopamine precursor levodopa to healthy participants in two experiments involving reaching movements. Levodopa has been shown to impair reward-based learning in cognitive tasks. Thus, we hypothesized that levodopa would selectively impair aspects of motor adaptation that depend on the reinforcement of rewarding actions. In the first experiment, participants performed two separate tasks in which adaptation was driven either by visual error-based feedback of the hand position or binary reward feedback. We used EEG to measure event-related potentials evoked by task feedback. We hypothesized that levodopa would specifically diminish adaptation and the neural responses to feedback in the reward learning task. However, levodopa did not affect motor adaptation in either task nor did it diminish event-related potentials elicited by reward outcomes. In the second experiment, participants learned to compensate for mechanical force field perturbations applied to the hand during reaching. Previous exposure to a particular force field can result in savings during subsequent adaptation to the same force field or interference during adaptation to an opposite force field. We hypothesized that levodopa would diminish savings and anterograde interference, as previous work suggests that these phenomena result from a reinforcement learning process. However, we found no reliable effects of levodopa. These results suggest that reward-based motor adaptation, savings, and interference may not depend on the same dopaminergic mechanisms that have been shown to be disrupted by levodopa during various cognitive tasks.NEW & NOTEWORTHY Motor adaptation relies on multiple processes including reinforcement of successful actions. Cognitive reinforcement learning is impaired by levodopa-induced disruption of dopamine function. We administered levodopa to healthy adults who participated in multiple motor adaptation tasks. We found no effects of levodopa on any component of motor adaptation. This suggests that motor adaptation may not depend on the same dopaminergic mechanisms as cognitive forms or reinforcement learning that have been shown to be impaired by levodopa.

Hierarchical motor adaptations negotiate failures during force field learning

Humans have the amazing ability to learn the dynamics of the body and environment to develop motor skills. Traditional motor studies using arm reaching paradigms have viewed this ability as the process of ‘internal model adaptation’. However, the behaviors have not been fully explored in the case when reaches fail to attain the intended target. Here we examined human reaching under two force fields types; one that induces failures (i.e., target errors), and the other that does not. Our results show the presence of a distinct failure-driven adaptation process that enables quick task success after failures, and before completion of internal model adaptation, but that can result in persistent changes to the undisturbed trajectory. These behaviors can be explained by considering a hierarchical interaction between internal model adaptation and the failure-driven adaptation of reach direction. Our findings suggest that movement failure is negotiated using hierarchical motor adaptations by humans.

How do we improve actions after a movement failure? Although negotiating movement failures is obviously crucial, previous motor-control studies have predominantly examined human movement adaptations in the absence of failures, and it remains unclear how failures affect subsequent movement adaptations. Here we examined this issue by developing a novel force field adaptation task where the hand movement during an arm reaching is perturbed by novel forces that induce a large target error, that is a failure. Our experimental observation and computational modeling show that, in addition to the popular ‘internal model learning’ process of motor adaptations, humans also utilize a ‘failure-negotiating’ process, that enables them to quickly improve movements in the presence of failure, even at the expense of increased arm trajectory deflections, which are subsequently reduced gradually with training after the achievement of the task success. Our results suggest that a hierarchical interaction between these two processes is a key for humans to negotiate movement failures.

Learning a reach trajectory based on binary reward feedback

Binary reward feedback on movement success is sufficient for learning some simple sensorimotor mappings in a reaching task, but not for some other tasks in which multiple kinematic factors contribute to performance. The critical condition for learning in more complex tasks remains unclear. Here, we investigate whether reward-based motor learning is possible in a multi-dimensional trajectory matching task and whether simplifying the task by providing feedback on one factor at a time ('factorized feedback') can improve learning. In two experiments, participants performed a trajectory matching task in which learning was measured as a reduction in the error. In Experiment 1, participants matched a straight trajectory slanted in depth. We factorized the task by providing feedback on the slant error, the length error, or on their composite. In Experiment 2, participants matched a curved trajectory, also slanted in depth. In this experiment, we factorized the feedback by providing feedback on the slant error, the curvature error, or on the integral difference between the matched and target trajectory. In Experiment 1, there was anecdotal evidence that participants learnt the multidimensional task. Factorization did not improve learning. In Experiment 2, there was anecdotal evidence the multidimensional task could not be learnt. We conclude that, within a complexity range, multiple kinematic factors can be learnt in parallel.

Task Feedback Processing Differs Between Young and Older Adults in Visuomotor Rotation Learning Despite Similar Initial Adaptation and Savings

Ageing has been suggested to affect sensorimotor adaptation by impairing explicit strategy use. Here we recorded electrophysiological (EEG) responses during visuomotor rotation in both young (n = 24) and older adults (n = 25), to investigate the neural processes that underpin putative age-related effects on adaptation. We measured the feedback related negativity (FRN) and the P3 in response to task-feedback, as electrophysiological markers of task error processing and outcome evaluation. The two age groups adapted similarly and showed comparable after effects and savings when re-exposed to the same perturbation several days after the initial session. Older adults, however, had less distinct EEG responses (i.e., reduced FRN amplitudes) to negative and positive task feedback. The P3 did not differ between age groups. Both young and older adults also showed a sustained late positivity following task feedback. Measured at the frontal electrode Fz, this sustained activity was negatively associated with both the amount of voluntary disengagement of explicit strategy and savings. In conclusion, despite preserved task performance, we find clear differences in neural responses to errors in older people, which suggests that there is a fundamental decline in this aspect of sensorimotor brain function with age.

Reinforcement regulates timing variability in thalamus

Learning reduces variability but variability can facilitate learning. This paradoxical relationship has made it challenging to tease apart sources of variability that degrade performance from those that improve it. We tackled this question in a context-dependent timing task requiring humans and monkeys to flexibly produce different time intervals with different effectors. We identified two opposing factors contributing to timing variability: slow memory fluctuation that degrades performance and reward-dependent exploratory behavior that improves performance. Signatures of these opposing factors were evident across populations of neurons in the dorsomedial frontal cortex (DMFC), DMFC-projecting neurons in the ventrolateral thalamus, and putative target of DMFC in the caudate. However, only in the thalamus were the performance-optimizing regulation of variability aligned to the slow performance-degrading memory fluctuations. These findings reveal how variability caused by exploratory behavior might help to mitigate other undesirable sources of variability and highlight a potential role for thalamocortical projections in this process.

The Psychology of Reaching: Action Selection, Movement Implementation, and Sensorimotor Learning

The study of motor planning and learning in humans has undergone a dramatic transformation in the 20 years since this journal's last review of this topic. The behavioral analysis of movement, the foundational approach for psychology, has been complemented by ideas from control theory, computer science, statistics, and, most notably, neuroscience. The result of this interdisciplinary approach has been a focus on the computational level of analysis, leading to the development of mechanistic models at the psychological level to explain how humans plan, execute, and consolidate skilled reaching movements. This review emphasizes new perspectives on action selection and motor planning, research that stands in contrast to the previously dominant representation-based perspective of motor programming, as well as an emerging literature highlighting the convergent operation of multiple processes in sensorimotor learning.

Reinforcement Signaling Can Be Used to Reduce Elements of Cerebellar Reaching Ataxia

Damage to the cerebellum causes a disabling movement disorder called ataxia, which is characterized by poorly coordinated movement. Arm ataxia causes dysmetria (over- or under-shooting of targets) with many corrective movements. As a result, people with cerebellar damage exhibit reaching movements with highly irregular and prolonged movement paths. Cerebellar patients are also impaired in error-based motor learning, which may impede rehabilitation interventions. However, we have recently shown that cerebellar patients can learn a simple reaching task using a binary reinforcement paradigm, in which feedback is based on participants' mean performance. Here, we present a pilot study that examined whether patients with cerebellar damage can use this reinforcement training to learn a more complex motor task-to decrease the path length of their reaches. We compared binary reinforcement training to a control condition of massed practice without reinforcement feedback. In both conditions, participants made target-directed reaches in 3-dimensional space while vision of their movement was occluded. In the reinforcement training condition, reaches with a path length below participants' mean were reinforced with an auditory stimulus at reach endpoint. We found that patients were able to use reinforcement signaling to significantly reduce their reach paths. Massed practice produced no systematic change in patients' reach performance. Overall, our results suggest that binary reinforcement training can improve reaching movements in patients with cerebellar damage and the benefit cannot be attributed solely to repetition or reduced visual control.

Neural Correlates of Reinforcement Learning in Mid-lateral Cerebellum

The role of the cerebellum in non-motor learning is poorly understood. Here, we investigated the activity of Purkinje cells (P-cells) in the mid-lateral cerebellum as the monkey learned to associate one arbitrary symbol with the movement of the left hand and another with the movement of the right hand. During learning, but not when the monkey had learned the association, the simple spike responses of P-cells reported the outcome of the animal's most recent decision without concomitant changes in other sensorimotor parameters such as hand movement, licking, or eye movement. At the population level, P-cells collectively maintained a memory of the most recent decision throughout the entire trial. As the monkeys learned the association, the magnitude of this reward-related error signal approached zero. Our results provide a major departure from the current understanding of cerebellar processing and have critical implications for cerebellum's role in cognitive control.

Domain-Specific Working Memory, But Not Dopamine-Related Genetic Variability, Shapes Reward-Based Motor Learning

The addition of rewarding feedback to motor learning tasks has been shown to increase the retention of learning, spurring interest in its possible utility for rehabilitation. However, motor tasks using rewarding feedback have repeatedly been shown to lead to great interindividual variability in performance. Understanding the causes of such variability is vital for maximizing the potential benefits of reward-based motor learning. Thus, using a large human cohort of both sexes (n = 241), we examined whether spatial (SWM), verbal, and mental rotation (RWM) working memory capacity and dopamine-related genetic profiles were associated with performance in two reward-based motor tasks. The first task assessed the participant's ability to follow a slowly shifting reward region based on hit/miss (binary) feedback. The second task investigated the participant's capacity to preserve performance with binary feedback after adapting to the rotation with full visual feedback. Our results demonstrate that higher SWM is associated with greater success and an enhanced capacity to reproduce a successful motor action, measured as change in reach angle following reward. In contrast, higher RWM was predictive of an increased propensity to express an explicit strategy when required to make large reach angle adjustments. Therefore, SWM and RWM were reliable, but dissociable, predictors of success during reward-based motor learning. Change in reach direction following failure was also a strong predictor of success rate, although we observed no consistent relationship with working memory. Surprisingly, no dopamine-related genotypes predicted performance. Therefore, working memory capacity plays a pivotal role in determining individual ability in reward-based motor learning.SIGNIFICANCE STATEMENT Reward-based motor learning tasks have repeatedly been shown to lead to idiosyncratic behaviors that cause varying degrees of task success. Yet, the factors determining an individual's capacity to use reward-based feedback are unclear. Here, we assessed a wide range of possible candidate predictors, and demonstrate that domain-specific working memory plays an essential role in determining individual capacity to use reward-based feedback. Surprisingly, genetic variations in dopamine availability were not found to play a role. This is in stark contrast with seminal work in the reinforcement and decision-making literature, which show strong and replicated effects of the same dopaminergic genes in decision-making. Therefore, our results provide novel insights into reward-based motor learning, highlighting a key role for domain-specific working memory capacity.

The influence of task outcome on implicit motor learning

Recent studies have demonstrated that task success signals can modulate learning during sensorimotor adaptation tasks, primarily through engaging explicit processes. Here, we examine the influence of task outcome on implicit adaptation, using a reaching task in which adaptation is induced by feedback that is not contingent on actual performance. We imposed an invariant perturbation (rotation) on the feedback cursor while varying the target size. In this way, the cursor either hit or missed the target, with the former producing a marked attenuation of implicit motor learning. We explored different computational architectures that might account for how task outcome information interacts with implicit adaptation. The results fail to support an architecture in which adaptation operates in parallel with a model-free operant reinforcement process. Rather, task outcome may serve as a gain on implicit adaptation or provide a distinct error signal for a second, independent implicit learning process.

Editorial note

This article has been through an editorial process in which the authors decide how to respond to the issues raised during peer review. The Reviewing Editor's assessment is that all the issues have been addressed (see decision letter).

Can patients with cerebellar disease switch learning mechanisms to reduce their adaptation deficits?

Systematic perturbations in motor adaptation tasks are primarily countered by learning from sensory-prediction errors, with secondary contributions from other learning processes. Despite the availability of these additional processes, particularly the use of explicit re-aiming to counteract observed target errors, patients with cerebellar degeneration are surprisingly unable to compensate for their sensory-prediction error deficits by spontaneously switching to another learning mechanism. We hypothesized that if the nature of the task was changed-by allowing vision of the hand, which eliminates sensory-prediction errors-patients could be induced to preferentially adopt aiming strategies to solve visuomotor rotations. To test this, we first developed a novel visuomotor rotation paradigm that provides participants with vision of their hand in addition to the cursor, effectively setting the sensory-prediction error signal to zero. We demonstrated in younger healthy control subjects that this promotes a switch to strategic re-aiming based on target errors. We then showed that with vision of the hand, patients with cerebellar degeneration could also switch to an aiming strategy in response to visuomotor rotations, performing similarly to age-matched participants (older controls). Moreover, patients could retrieve their learned aiming solution after vision of the hand was removed (although they could not improve beyond what they retrieved), and retain it for at least 1 year. Both patients and older controls, however, exhibited impaired overall adaptation performance compared to younger healthy controls (age 18-33 years), likely due to age-related reductions in spatial and working memory. Patients also failed to generalize, i.e. they were unable to adopt analogous aiming strategies in response to novel rotations. Hence, there appears to be an inescapable obligatory dependence on sensory-prediction error-based learning-even when this system is impaired in patients with cerebellar disease. The persistence of sensory-prediction error-based learning effectively suppresses a switch to target error-based learning, which perhaps explains the unexpectedly poor performance by patients with cerebellar degeneration in visuomotor adaptation tasks.

The gradient of the reinforcement landscape influences sensorimotor learning

Consideration of previous successes and failures is essential to mastering a motor skill. Much of what we know about how humans and animals learn from such reinforcement feedback comes from experiments that involve sampling from a small number of discrete actions. Yet, it is less understood how we learn through reinforcement feedback when sampling from a continuous set of possible actions. Navigating a continuous set of possible actions likely requires using gradient information to maximize success. Here we addressed how humans adapt the aim of their hand when experiencing reinforcement feedback that was associated with a continuous set of possible actions. Specifically, we manipulated the change in the probability of reward given a change in motor action-the reinforcement gradient-to study its influence on learning. We found that participants learned faster when exposed to a steep gradient compared to a shallow gradient. Further, when initially positioned between a steep and a shallow gradient that rose in opposite directions, participants were more likely to ascend the steep gradient. We introduce a model that captures our results and several features of motor learning. Taken together, our work suggests that the sensorimotor system relies on temporally recent and spatially local gradient information to drive learning.

Neural signatures of reward and sensory error feedback processing in motor learning

At least two distinct processes have been identified by which motor commands are adapted according to movement-related feedback: reward-based learning and sensory error-based learning. In sensory error-based learning, mappings between sensory targets and motor commands are recalibrated according to sensory error feedback. In reward-based learning, motor commands are associated with subjective value, such that successful actions are reinforced. We designed two tasks to isolate reward- and sensory error-based motor adaptation, and we used electroencephalography in humans to identify and dissociate the neural correlates of reward and sensory error feedback processing. We designed a visuomotor rotation task to isolate sensory error-based learning that was induced by altered visual feedback of hand position. In a reward learning task, we isolated reward-based learning induced by binary reward feedback that was decoupled from the visual target. A fronto-central event-related potential called the feedback-related negativity (FRN) was elicited specifically by binary reward feedback but not sensory error feedback. A more posterior component called the P300 was evoked by feedback in both tasks. In the visuomotor rotation task, P300 amplitude was increased by sensory error induced by perturbed visual feedback and was correlated with learning rate. In the reward learning task, P300 amplitude was increased by reward relative to nonreward and by surprise regardless of feedback valence. We propose that during motor adaptation the FRN specifically reflects a reward-based learning signal whereas the P300 reflects feedback processing that is related to adaptation more generally. NEW & NOTEWORTHY We studied the event-related potentials evoked by feedback stimuli during motor adaptation tasks that isolate reward- and sensory error-based learning mechanisms. We found that the feedback-related negativity was specifically elicited by binary reward feedback, whereas the P300 was observed in both tasks. These results reveal neural processes associated with different learning mechanisms and elucidate which classes of errors, from a computational standpoint, elicit the feedback-related negativity and P300.

Task errors contribute to implicit aftereffects in sensorimotor adaptation

Perturbations of sensory feedback evoke sensory prediction errors (discrepancies between predicted and actual sensory outcomes of movements), and reward prediction errors (discrepancies between predicted rewards and actual rewards). When our task is to hit a target, we expect to succeed in hitting the target, and so we experience a reward prediction error if the perturbation causes us to miss it. These discrepancies between intended task outcomes and actual task outcomes, termed "task errors," are thought to drive the use of strategic processes to restore success, although their role is incompletely understood. Here, as participants adapted to a 30° rotation of cursor feedback representing hand position, we investigated the role of task errors in sensorimotor adaptation: during target-reaching, we either removed task errors by moving the target mid-movement to align with cursor feedback of hand position, or enforced task error by moving the target away from the cursor feedback of hand position, by 20-30° randomly (clockwise in half the trials, counterclockwise in half the trials). Removing task errors not only reduced the extent of adaptation during exposure to the perturbation, but also reduced the amount of post-adaptation aftereffects that persisted despite explicit knowledge of the perturbation removal. Hence, task errors contribute to implicit adaptation resulting from sensory prediction errors. This suggests that the system which predicts the sensory consequences of actions via exposure to sensory prediction errors is also sensitive to reward prediction errors.

Somatosensory working memory in human reinforcement-based motor learning

Recent studies using visuomotor adaptation and sequence learning tasks have assessed the involvement of working memory in the visuospatial domain. The capacity to maintain previously performed movements in working memory is perhaps even more important in reinforcement-based learning to repeat accurate movements and avoid mistakes. Using this kind of task in the present work, we tested the relationship between somatosensory working memory and motor learning. The first experiment involved separate memory and motor learning tasks. In the memory task, the participant's arm was displaced in different directions by a robotic arm, and the participant was asked to judge whether a subsequent test direction was one of the previously presented directions. In the motor learning task, participants made reaching movements to a hidden visual target and were provided with positive feedback as reinforcement when the movement ended in the target zone. It was found that participants that had better somatosensory working memory showed greater motor learning. In a second experiment, we designed a new task in which learning and working memory trials were interleaved, allowing us to study participants' memory for movements they performed as part of learning. As in the first experiment, we found that participants with better somatosensory working memory also learned more. Moreover, memory performance for successful movements was better than for movements that failed to reach the target. These results suggest that somatosensory working memory is involved in reinforcement motor learning and that this memory preferentially keeps track of reinforced movements. NEW & NOTEWORTHY The present work examined somatosensory working memory in reinforcement-based motor learning. Working memory performance was reliably correlated with the extent of learning. With the use of a paradigm in which learning and memory trials were interleaved, memory was assessed for movements performed during learning. Movements that received positive feedback were better remembered than movements that did not. Thus working memory does not track all movements equally but is biased to retain movements that were rewarded.

Uncertainty modulated exploration in the trade-off between sensing and acting

Many sensorimotor activities have a time constraint for successful completion. In this case, any time devoted to sensory processing is at the expense of time available for motor execution. Earlier studies have explored how this competition between sensory processing and motor execution is resolved by using experimental designs that segregate the sensing and acting phase of the task. It was found that participants switch from the sensing to the acting stage such that the overall (sensorimotor) uncertainty in the outcome of the task is minimized. An unexplained observation in these studies is the substantial variability in switching times. We investigated the variability in switching time by correlating it with the underlying sensorimotor uncertainty. To this end, we used a modified version of the virtual ball catching paradigm proposed by Faisal & Wolpert (2009). Subjects were instructed to catch a ball, but as soon as they initiated their movement the ball disappeared. We extended the range of horizontal velocities and used two different start positions of the ball to quantify the dependence of sensory uncertainty on ball velocity. Velocity dependence of sensory uncertainty allowed us to manipulate sensory uncertainty and hence the sensorimotor uncertainty. We found that the variability in switching times is correlated with two factors. Firstly, variability in switching times is greater when variation in switching time has only minimal effects on sensorimotor uncertainty. Secondly, variability in switching times is greater when the sensorimotor uncertainty is larger. Our results suggest that the variability in switching time reflects an uncertainty-driven exploratory process.