Neural signatures of reward and sensory error feedback processing in motor learning

Spatio-temporal modulation of cortical activity during motor deadaptation depends on the feedback of task-related error

Motor adaptations are responsible for recalibrating actions and facilitating the achievement of goals in a constantly changing environment. Once consolidated, the decay of motor adaptation is a process affected by available sensory information during deadaptation. However, the cortical response to task error feedback during the deadaptation phase has received little attention. Here, we explored changes in brain cortical responses due to feedback of task-related error during deadaptation. Twelve healthy volunteers were recruited for the study. Right hand movement and EEG were recorded during repetitive trials of a hand reaching movement. A visuomotor rotation of 30° was introduced to induce motor adaptation. Volunteers participated in two experimental sessions organized in baseline, adaptation, and deadaptation blocks. In the deadaptation block, the visuomotor rotation was removed, and visual feedback was only provided in one session. Performance was quantified using angle end-point error, averaged speed, and movement onset time. A non-parametric spatiotemporal cluster-level permutation test was used to analyze the EEG recordings. During deadaptation, participants experienced a greater error reduction when feedback of the cursor was provided. The EEG responses showed larger activity in the left centro-frontal parietal areas during the deadaptation block when participants received feedback, as opposed to when they did not receive feedback. Centrally distributed clusters were found for the adaptation and deadaptation blocks in the absence of visual feedback. The results suggest that visual feedback of the task-related error activates cortical areas related to performance monitoring, depending on the accessible sensory information.

Oculomotor behavior can be adjusted on the basis of artificial feedback signals indicating externally caused errors

Whether a saccade is accurate and has reached the target cannot be evaluated during its execution, but relies on post-saccadic feedback. If the eye has missed the target object, a secondary corrective saccade has to be made to align the fovea with the target. If a systematic post-saccadic error occurs, adaptive changes to the oculomotor behavior are made, such as shortening or lengthening the saccade amplitude. Systematic post-saccadic errors are typically attributed internally to erroneous motor commands. The corresponding adaptive changes to the motor command reduce the error and the need for secondary corrective saccades, and, in doing so, restore accuracy and efficiency. However, adaptive changes to the oculomotor behavior also occur if a change in saccade amplitude is beneficial for task performance, or if it is rewarded. Oculomotor learning thus is more complex than reducing a post-saccadic position error. In the current study, we used a novel oculomotor learning paradigm and investigated whether human participants are able to adapt their oculomotor behavior to improve task performance even when they attribute the error externally. The task was to indicate the intended target object among several objects to a simulated human-machine interface by making eye movements. The participants were informed that the system itself could make errors. The decoding process depended on a distorted landing point of the saccade, resulting in decoding errors. Two different types of visual feedback were added to the post-saccadic scene and we compared how participants used the different feedback types to adjust their oculomotor behavior to avoid errors. We found that task performance improved over time, regardless of the type of feedback. Thus, error feedback from the simulated human-machine interface was used for post-saccadic error evaluation. This indicates that 1) artificial visual feedback signals and 2) externally caused errors might drive adaptive changes to oculomotor behavior.

Exploration biases forelimb reaching strategies

The brain can generate actions, such as reaching to a target, using different movement strategies. We investigate how such strategies are learned in a task where perched head-fixed mice learn to reach to an invisible target area from a set start position using a joystick. This can be achieved by learning to move in a specific direction or to a specific endpoint location. As mice learn to reach the target, they refine their variable joystick trajectories into controlled reaches, which depend on the sensorimotor cortex. We show that individual mice learned strategies biased to either direction- or endpoint-based movements. This endpoint/direction bias correlates with spatial directional variability with which the workspace was explored during training. Model-free reinforcement learning agents can generate both strategies with similar correlation between variability during training and learning bias. These results provide evidence that reinforcement of individual exploratory behavior during training biases the reaching strategies that mice learn.

The effect of reward on motor learning: different stage, different effect

Motor learning is a prominent and extensively studied subject in rehabilitation following various types of neurological disorders. Motor repair and rehabilitation often extend over months and years post-injury with a slow pace of recovery, particularly affecting the fine movements of the distal extremities. This extended period can diminish the motivation and persistence of patients, a facet that has historically been overlooked in motor learning until recent years. Reward, including monetary compensation, social praise, video gaming, music, and virtual reality, is currently garnering heightened attention for its potential to enhance motor motivation and improve function. Numerous studies have examined the effects and attempted to explore potential mechanisms in various motor paradigms, yet they have yielded inconsistent or even contradictory results and conclusions. A comprehensive review is necessary to summarize studies on the effects of rewards on motor learning and to deduce a central pattern from these existing studies. Therefore, in this review, we initially outline a framework of motor learning considering two major types, two major components, and three stages. Subsequently, we summarize the effects of rewards on different stages of motor learning within the mentioned framework and analyze the underlying mechanisms at the level of behavior or neural circuit. Reward accelerates learning speed and enhances the extent of learning during the acquisition and consolidation stages, possibly by regulating the balance between the direct and indirect pathways (activating more D1-MSN than D2-MSN) of the ventral striatum and by increasing motor dynamics and kinematics. However, the effect varies depending on several experimental conditions. During the retention stage, there is a consensus that reward enhances both short-term and long-term memory retention in both types of motor learning, attributed to the LTP learning mechanism mediated by the VTA-M1 dopaminergic projection. Reward is a promising enhancer to bolster waning confidence and motivation, thereby increasing the efficiency of motor learning and rehabilitation. Further exploration of the circuit and functional connections between reward and the motor loop may provide a novel target for neural modulation to promote motor behavior.

NSF DARE-transforming modeling in neurorehabilitation: a patient-in-the-loop framework

In 2023, the National Science Foundation (NSF) and the National Institute of Health (NIH) brought together engineers, scientists, and clinicians by sponsoring a conference on computational modelling in neurorehabiilitation. To facilitate multidisciplinary collaborations and improve patient care, in this perspective piece we identify where and how computational modelling can support neurorehabilitation. To address the where, we developed a patient-in-the-loop framework that uses multiple and/or continual measurements to update diagnostic and treatment model parameters, treatment type, and treatment prescription, with the goal of maximizing clinically-relevant functional outcomes. This patient-in-the-loop framework has several key features: (i) it includes diagnostic and treatment models, (ii) it is clinically-grounded with the International Classification of Functioning, Disability and Health (ICF) and patient involvement, (iii) it uses multiple or continual data measurements over time, and (iv) it is applicable to a range of neurological and neurodevelopmental conditions. To address the how, we identify state-of-the-art and highlight promising avenues of future research across the realms of sensorimotor adaptation, neuroplasticity, musculoskeletal, and sensory & pain computational modelling. We also discuss both the importance of and how to perform model validation, as well as challenges to overcome when implementing computational models within a clinical setting. The patient-in-the-loop approach offers a unifying framework to guide multidisciplinary collaboration between computational and clinical stakeholders in the field of neurorehabilitation.

Dorsomedial frontal cortex damage impairs error-based, but not reinforcement-based motor learning in humans

We adapt our movements to new and changing environments through multiple processes. Sensory error-based learning counteracts environmental perturbations that affect the sensory consequences of movements. Sensory errors also cause the upregulation of reflexes and muscle co-contraction. Reinforcement-based learning enhances the selection of movements that produce rewarding outcomes. Although some findings have identified dissociable neural substrates of sensory error- and reinforcement-based learning, correlative methods have implicated dorsomedial frontal cortex in both. Here, we tested the causal contributions of dorsomedial frontal to adaptive motor control, studying people with chronic damage to this region. Seven human participants with focal brain lesions affecting the dorsomedial frontal and 20 controls performed a battery of arm movement tasks. Three experiments tested: (i) the upregulation of visuomotor reflexes and muscle co-contraction in response to unpredictable mechanical perturbations, (ii) sensory error-based learning in which participants learned to compensate predictively for mechanical force-field perturbations, and (iii) reinforcement-based motor learning based on binary feedback in the absence of sensory error feedback. Participants with dorsomedial frontal damage were impaired in the early stages of force field adaptation, but performed similarly to controls in all other measures. These results provide evidence for a specific and selective causal role for the dorsomedial frontal in sensory error-based learning.

Explicit learning based on reward prediction error facilitates agile motor adaptations

Error based motor learning can be driven by both sensory prediction error and reward prediction error. Learning based on sensory prediction error is termed sensorimotor adaptation, while learning based on reward prediction error is termed reward learning. To investigate the characteristics and differences between sensorimotor adaptation and reward learning, we adapted a visuomotor paradigm where subjects performed arm movements while presented with either the sensory prediction error, signed end-point error, or binary reward. Before each trial, perturbation indicators in the form of visual cues were presented to inform the subjects of the presence and direction of the perturbation. To analyse the interconnection between sensorimotor adaptation and reward learning, we designed a computational model that distinguishes between the two prediction errors. Our results indicate that subjects adapted to novel perturbations irrespective of the type of prediction error they received during learning, and they converged towards the same movement patterns. Sensorimotor adaptations led to a pronounced aftereffect, while adaptation based on reward consequences produced smaller aftereffects suggesting that reward learning does not alter the internal model to the same degree as sensorimotor adaptation. Even though all subjects had learned to counteract two different perturbations separately, only those who relied on explicit learning using reward prediction error could timely adapt to the randomly changing perturbation. The results from the computational model suggest that sensorimotor and reward learning operate through distinct adaptation processes and that only sensorimotor adaptation changes the internal model, whereas reward learning employs explicit strategies that do not result in aftereffects. Additionally, we demonstrate that when humans learn motor tasks, they utilize both learning processes to successfully adapt to the new environments.

Feedback processing in cognitive and motor tasks: A meta-analysis on the feedback-related negativity

For motor learning, the processing of behavioral outcomes is of high significance. The feedback-related negativity (FRN) is an event-related potential, which is often described as a correlate of the reward prediction error in reinforcement learning. The number of studies examining the FRN in motor tasks is increasing. This meta-analysis summarizes the component in the motor domain and compares it to the cognitive domain. Therefore, a data set of a previous meta-analysis in the cognitive domain that comprised 47 studies  was reanalyzed and compared to additional 25 studies of the motor domain. Further, a moderator analysis for the studies in the motor domain was conducted. The FRN amplitude was higher in the motor domain than in the cognitive domain. This might be related to a higher task complexity and a higher feedback ambiguity of motor tasks. The FRN latency was shorter in the motor domain than in the cognitive domain. Given that sensory information can be used as an external feedback predictor prior to the presentation of the final feedback, reward processing in the motor domain may have been faster and reduced the FRN latency. The moderator variable analysis revealed that the feedback modality influenced the FRN latency, with shorter FRN latencies after bimodal than after visual feedback. Processing of outcome feedback seems to share basic principles in both domains; however, differences exist and should be considered in FRN studies. Future research is motivated to scrutinize the effects of bimodal feedback and other moderators within the motor domain.

Combined low-frequency brain oscillatory activity and behavior predict future errors in human motor skill

Human skills are composed of sequences of individual actions performed with utmost precision. When occasional errors occur, they may have serious consequences, for example, when pilots are manually landing a plane. In such cases, the ability to predict an error before it occurs would clearly be advantageous. Here, we asked whether it is possible to predict future errors in a keyboard procedural human motor skill. We report that prolonged keypress transition times (KTTs), reflecting slower speed, and anomalous delta-band oscillatory activity in cingulate-entorhinal-precuneus brain regions precede upcoming errors in skill. Combined anomalous low-frequency activity and prolonged KTTs predicted up to 70% of future errors. Decoding strength (posterior probability of error) increased progressively approaching the errors. We conclude that it is possible to predict future individual errors in skill sequential performance.

Sparse Logistic Regression-Based EEG Channel Optimization Algorithm for Improved Universality across Participants

Electroencephalogram (EEG) channel optimization can reduce redundant information and improve EEG decoding accuracy by selecting the most informative channels. This article aims to investigate the universality regarding EEG channel optimization in terms of how well the selected EEG channels can be generalized to different participants. In particular, this study proposes a sparse logistic regression (SLR)-based EEG channel optimization algorithm using a non-zero model parameter ranking method. The proposed channel optimization algorithm was evaluated in both individual analysis and group analysis using the raw EEG data, compared with the conventional channel selection method based on the correlation coefficients (CCS). The experimental results demonstrate that the SLR-based EEG channel optimization algorithm not only filters out most redundant channels (filters 75-96.9% of channels) with a 1.65-5.1% increase in decoding accuracy, but it can also achieve a satisfactory level of decoding accuracy in the group analysis by employing only a few (2-15) common EEG electrodes, even for different participants. The proposed channel optimization algorithm can realize better universality for EEG decoding, which can reduce the burden of EEG data acquisition and enhance the real-world application of EEG-based brain-computer interface (BCI).

Exploration biases how forelimb reaches to a spatial target are learned

The brain can learn to generate actions, such as reaching to a target, using different movement strategies. Understanding how different variables bias which strategies are learned to produce such a reach is important for our understanding of the neural bases of movement. Here we introduce a novel spatial forelimb target task in which perched head-fixed mice learn to reach to a circular target area from a set start position using a joystick. These reaches can be achieved by learning to move into a specific direction or to a specific endpoint location. We find that mice gradually learn to successfully reach the covert target. With time, they refine their initially exploratory complex joystick trajectories into controlled targeted reaches. The execution of these controlled reaches depends on the sensorimotor cortex. Using a probe test with shifting start positions, we show that individual mice learned to use strategies biased to either direction or endpoint-based movements. The degree of endpoint learning bias was correlated with the spatial directional variability with which the workspace was explored early in training. Furthermore, we demonstrate that reinforcement learning model agents exhibit a similar correlation between directional variability during training and learned strategy. These results provide evidence that individual exploratory behavior during training biases the control strategies that mice use to perform forelimb covert target reaches.

Bridging event-related potentials with behavioral studies in motor learning

Behavioral approaches and electrophysiology in understanding human sensorimotor systems have both yielded substantial advancements in past decades. In fact, behavioral neuroscientists have found that motor learning involves the two distinct processes of the implicit and the explicit. Separately, they have also distinguished two kinds of errors that drive motor learning: sensory prediction error and task error. Scientists in electrophysiology, in addition, have discovered two motor-related, event-related potentials (ERPs): error-related negativity (ERN), and feedback-related negativity (FRN). However, there has been a lack of interchange between the two lines of research. This article, therefore, will survey through the literature in both directions, attempting to establish a bridge between these two fruitful lines of research.

Relationship between reward-related evoked potentials and real-world motivation in older people living with human immunodeficiency virus

Apathy, a clinical disorder characterized by low motivation, is prevalent in people living with Human Immunodeficiency Virus (HIV). It affects mental and physical health-related quality-of-life, medication adherence, and is associated with cognitive decline. However, the causes of apathy and the underlying brain mechanisms in HIV are unknown. Brain responses to reward may be relevant to understanding apathy and might serve as biomarkers for diagnosis or treatment response. Electroencephalogram (EEG) responses to gain and loss feedback in simple guessing tasks have been related to apathy in neurodegenerative conditions and healthy individuals. The primary aim of this study is to contribute evidence regarding the relationship between two EEG correlates of reward processing, the Reward Positivity, and the Feedback-P300, and real-world motivated behavior indicated by self-reported hours engaged in goal-directed leisure activities per week, in older individuals with well-controlled HIV infection. High-density EEG was collected from 75 participants while they performed a guessing task with gain or loss feedback. We found that a later component of reward processing, the Feedback-P300, was related to real-world engagement, while the earlier Reward Positivity was not. The Feedback-P300 measured with EEG holds promise as a biomarker for motivated behavior in older people living with HIV. These findings lay the groundwork for a better understanding of the neurobiology of apathy in this condition.

Distinct Neural Signatures of Outcome Monitoring After Selection and Execution Errors

Losing a point in tennis could result from poor shot selection or faulty stroke execution. To explore how the brain responds to these different types of errors, we examined feedback-locked EEG activity while participants completed a modified version of a standard three-armed bandit probabilistic reward task. Our task framed unrewarded outcomes as the result of either errors of selection or errors of execution. We examined whether amplitude of a medial frontal negativity (the feedback-related negativity [FRN]) was sensitive to the different forms of error attribution. Consistent with previous reports, selection errors elicited a large FRN relative to rewards, and amplitude of this signal correlated with behavioral adjustment after these errors. A different pattern was observed in response to execution errors. These outcomes produced a larger FRN, a frontocentral attenuation in activity preceding this component, and a subsequent enhanced error positivity in parietal sites. Notably, the only correlations with behavioral adjustment were with the early frontocentral attenuation and amplitude of the parietal signal; FRN differences between execution errors and rewarded trials did not correlate with subsequent changes in behavior. Our findings highlight distinct neural correlates of selection and execution error processing, providing insight into how the brain responds to the different classes of error that determine future action.

Using EEG to study sensorimotor adaptation

Sensorimotor adaptation, or the capacity to flexibly adapt movements to changes in the body or the environment, is crucial to our ability to move efficiently in a dynamic world. The field of sensorimotor adaptation is replete with rigorous behavioural and computational methods, which support strong conceptual frameworks. An increasing number of studies have combined these methods with electroencephalography (EEG) to unveil insights into the neural mechanisms of adaptation. We review these studies: discussing EEG markers of adaptation in the frequency and the temporal domain, EEG predictors for successful adaptation and how EEG can be used to unmask latent processes resulting from adaptation, such as the modulation of spatial attention. With its high temporal resolution, EEG can be further exploited to deepen our understanding of sensorimotor adaptation.

Observing errors in a combination of error and correct models favors observational motor learning

Background

Imitative learning is highly effective from infancy to old age; however, little is known about the effects of observing errors during imitative learning. This study aimed to examine how observing errors affected imitative learning performance to maximize its effect.

Methods

In the pre-training session, participants were instructed to pinch at a target force (8 N) with auditory feedback regarding generated force while they watched videos of someone pinching a sponge at the target force. In the pre-test, participants pinched at the target force and did not view a model or receive auditory feedback. In Experiment 1, in the main training session, participants imitated models while they watched videos of pinching at either the incorrect force (error-mixed condition) or target force (correct condition). Then, the exact force generated was measured without receiving auditory feedback or viewing a model. In Experiment 2, using the same procedures, newly recruited participants watched videos of pinching at incorrect forces (4 and 24 N) as the error condition and the correct force as the correct condition.

Results

In Experiment 1, the average force was closer to the target force in the error-mixed condition than in the correct condition. In Experiment 2, the average force in the correct condition was closer to the target force than in the error condition.

Conclusion

Our findings indicated that observing error actions combined with correct actions affected imitation motor learning positively as error actions contained information on things to avoid in the target action. It provides further information to enhance imitative learning in mixed conditions compared to that with correct action alone.

Clustering analysis of movement kinematics in reinforcement learning

Reinforcement learning has been used as an experimental model of motor skill acquisition, where at times movements are successful and thus reinforced. One fundamental problem is to understand how humans select exploration over exploitation during learning. The decision could be influenced by factors such as task demands and reward availability. In this study, we applied a clustering algorithm to examine how a change in the accuracy requirements of a task affected the choice of exploration over exploitation. Participants made reaching movements to an unseen target using a planar robot arm and received reward after each successful movement. For one group of participants, the width of the hidden target decreased after every other training block. For a second group, it remained constant. The clustering algorithm was applied to the kinematic data to characterize motor learning on a trial-to-trial basis as a sequence of movements, each belonging to one of the identified clusters. By the end of learning, movement trajectories across all participants converged primarily to a single cluster with the greatest number of successful trials. Within this analysis framework, we defined exploration and exploitation as types of behavior in which two successive trajectories belong to different or similar clusters, respectively. The frequency of each mode of behavior was evaluated over the course of learning. It was found that by reducing the target width, participants used a greater variety of different clusters and displayed more exploration than exploitation. Excessive exploration relative to exploitation was found to be detrimental to subsequent motor learning.

NEW & NOTEWORTHY The choice of exploration versus exploitation is a fundamental problem in learning new motor skills through reinforcement. In this study, we employed a data-driven approach to characterize movements on a trial-by-trial basis with an unsupervised clustering algorithm. Using this technique, we found that changes in task demands and, in particular, in the required accuracy of movements, influenced the ratio of exploration to exploitation. This analysis framework provides an attractive tool to investigate mechanisms of explorative and exploitative behavior while studying motor learning.

Individual differences in error-related frontal midline theta activity during visuomotor adaptation

Post-feedback frontal midline EEG activity has been found to correlate with error magnitude during motor adaptation. However, the role of this neuronal activity remains to be elucidated. It has been hypothesized that post-feedback frontal midline activity may represent a prediction error, which in turn may be directly related to the adaptation process or to an unspecific orienting response. To address these hypotheses, we replicated a previous visuomotor adaptation experiment with very small perturbations, likely to invoke implicit adaptation, in a new group of 60 participants and combined it with EEG recordings. We found error-related peaks in the frontal midline electrodes in the time domain. However, these were best understood as modulations of frontal midline theta activity (FMT, 4-8 Hz). Trial-level differences in FMT correlated with error magnitude. This correlation was robust even for very small errors as well as in the absence of imposed perturbations, indicating that FMT does not depend on explicit or strategic re-aiming. Within participants, trial-level differences in FMT were not related to between-trial error corrections. Between participants, individual differences in FMT-error-sensitivity did not predict differences in adaptation rate. Taken together, these results imply that FMT does not drive implicit motor adaptation. Finally, individual differences in FMT-error-sensitivity negatively correlate to motor execution noise. This suggests that FMT reflects saliency: larger execution noise means a larger standard deviation of errors so that a fixed error magnitude is less salient. In conclusion, this study suggests that frontal midline theta activity represents a saliency signal and does not directly drive motor adaptation.

Never too little: Grip and lift forces following probabilistic weight cues in patients with writer's cramp

OBJECTIVE

Planning of voluntary object-related movements requires the estimation of the most probable object properties. We investigated how 14 writer's cramp (WC) patients compared to 14 controls use probabilistic weight cues in a serial grip-lift task.

METHODS

In every grip-lift trial, an object of either light, medium or heavy weight had to be grasped and lifted after a visual cue gave a probabilistic prediction of the object weights (e.g. 32.5% light, 67.5% medium, 0 % heavy). We determined peak (1) grip force GF, (2) load force LF, (3) grip force rate GFR, (4) load force rate LFR, while we registered brain activity with functional magnetic resonance imaging.

RESULTS

In both groups, GFR, LFR and GF increased when a higher probability of heavy weights was announced. When a higher probability of light weights was indicated, controls reduced GFR, LFR and GF, while WC patients did not downscale their forces. There were no inter-group differences in blood oxygenation level dependent activation.

CONCLUSIONS

WC patients could not utilize the decision range in motor planning and adjust their force in a probabilistic cued fine motor task.

SIGNIFICANCE

The results support the pathophysiological model of a hyperfunctional dopamine dependent direct basal ganglia pathway in WC.

Cortical preparatory activity during motor learning reflects visuomotor retention deficits after punishment feedback

Previous studies have shown that reinforcement-based motor learning requires the brain to process feedback-related information after movement execution. However, whether reinforcement feedback changes how the brain processes motor preparation before movement execution is unclear. By using electroencephalography (EEG), this study investigates whether reinforcement feedback changes cortical preparatory activity to modulate motor learning and memory. Human subjects were divided in three groups [reward, punishment, control] to perform a visuomotor rotation task under different conditions that assess adaptation (learning) and retention (memory) during the task. Reinforcement feedback was provided in the form of points after each trial that signaled monetary gains (reward) or losses (punishment). EEG was utilized to evaluate the amplitude of movement readiness potentials (MRPs) at the beginning of each trial for each group during the adaptation and retention conditions of the task. The results show that punishment feedback significantly decreased MRPs amplitude during both task conditions compared to Reward and Control groups. Moreover, the punishment-related decrease in MRPs amplitude paralleled decreases in motor performance during the retention but not the adaptation condition. No changes in MRPs or motor performance were observed in the Reward group. These results support the idea that reinforcement feedback modulates motor preparation and suggest that changes in cortical preparatory activity contribute to the visuomotor retention deficits observed after punishment feedback.

Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them

When learning a movement based on binary success information, one is more variable following failure than following success. Theoretically, the additional variability post-failure might reflect exploration of possibilities to obtain success. When average behavior is changing (as in learning), variability can be estimated from differences between subsequent movements. Can one estimate exploration reliably from such trial-to-trial changes when studying reward-based motor learning? To answer this question, we tried to reconstruct the exploration underlying learning as described by four existing reward-based motor learning models. We simulated learning for various learner and task characteristics. If we simply determined the additional change post-failure, estimates of exploration were sensitive to learner and task characteristics. We identified two pitfalls in quantifying exploration based on trial-to-trial changes. Firstly, performance-dependent feedback can cause correlated samples of motor noise and exploration on successful trials, which biases exploration estimates. Secondly, the trial relative to which trial-to-trial change is calculated may also contain exploration, which causes underestimation. As a solution, we developed the additional trial-to-trial change (ATTC) method. By moving the reference trial one trial back and subtracting trial-to-trial changes following specific sequences of trial outcomes, exploration can be estimated reliably for the three models that explore based on the outcome of only the previous trial. Since ATTC estimates are based on a selection of trial sequences, this method requires many trials. In conclusion, if exploration is a binary function of previous trial outcome, the ATTC method allows for a model-free quantification of exploration.

Subsequent memory effects on event-related potentials in associative fear learning

Studies of human fear learning suggest that a reliable discrimination between safe and threatening stimuli is important for survival and mental health. In the current study, we applied the subsequent memory paradigm in order to identify neurophysiological correlates of successful threat and safety learning. We recorded event-related potentials, while participants incidentally learned associations between multiple neutral faces and an aversive outcome [unconditioned stimulus (US)/conditioned stimulus (CS)+] or no outcome (noUS/CS-). We found that an enhanced late positive potential (LPP) to both CS+ and CS- during learning predicted subsequent memory. A quadratic relationship between LPP and confidence in memory indicates a possible role in both correct and false fear memory. Importantly, the P300 to the omission of the US (following CS-) was enhanced for remembered CS-, while there was a positive correlation between P300 amplitude to both US occurrence and omission and individual memory performance. A following re-exposure phase indicated that memory was indeed related to subjective fear of the CS+/CS-. These results highlight the importance of cognitive resource allocation to both threat and safety for the acquisition of fear and suggest a potential role of the P300 to US omission as an electrophysiological marker of successful safety learning.

Reward boosts reinforcement-based motor learning

Besides relying heavily on sensory and reinforcement feedback, motor skill learning may also depend on the level of motivation experienced during training. Yet, how motivation by reward modulates motor learning remains unclear. In 90 healthy subjects, we investigated the net effect of motivation by reward on motor learning while controlling for the sensory and reinforcement feedback received by the participants. Reward improved motor skill learning beyond performance-based reinforcement feedback. Importantly, the beneficial effect of reward involved a specific potentiation of reinforcement-related adjustments in motor commands, which concerned primarily the most relevant motor component for task success and persisted on the following day in the absence of reward. We propose that the long-lasting effects of motivation on motor learning may entail a form of associative learning resulting from the repetitive pairing of the reinforcement feedback and reward during training, a mechanism that may be exploited in future rehabilitation protocols.

Null effects of levodopa on reward- and error-based motor adaptation, savings, and anterograde interference

Dopamine signaling is thought to mediate reward-based learning. We tested for a role of dopamine in motor adaptation by administering the dopamine precursor levodopa to healthy participants in two experiments involving reaching movements. Levodopa has been shown to impair reward-based learning in cognitive tasks. Thus, we hypothesized that levodopa would selectively impair aspects of motor adaptation that depend on the reinforcement of rewarding actions. In the first experiment, participants performed two separate tasks in which adaptation was driven either by visual error-based feedback of the hand position or binary reward feedback. We used EEG to measure event-related potentials evoked by task feedback. We hypothesized that levodopa would specifically diminish adaptation and the neural responses to feedback in the reward learning task. However, levodopa did not affect motor adaptation in either task nor did it diminish event-related potentials elicited by reward outcomes. In the second experiment, participants learned to compensate for mechanical force field perturbations applied to the hand during reaching. Previous exposure to a particular force field can result in savings during subsequent adaptation to the same force field or interference during adaptation to an opposite force field. We hypothesized that levodopa would diminish savings and anterograde interference, as previous work suggests that these phenomena result from a reinforcement learning process. However, we found no reliable effects of levodopa. These results suggest that reward-based motor adaptation, savings, and interference may not depend on the same dopaminergic mechanisms that have been shown to be disrupted by levodopa during various cognitive tasks.NEW & NOTEWORTHY Motor adaptation relies on multiple processes including reinforcement of successful actions. Cognitive reinforcement learning is impaired by levodopa-induced disruption of dopamine function. We administered levodopa to healthy adults who participated in multiple motor adaptation tasks. We found no effects of levodopa on any component of motor adaptation. This suggests that motor adaptation may not depend on the same dopaminergic mechanisms as cognitive forms or reinforcement learning that have been shown to be impaired by levodopa.

Differential Theta-Band Signatures of the Anterior Cingulate and Motor Cortices During Seated Locomotor Perturbations

Quantifying motor and cortical responses to perturbations during seated locomotor tasks such as recumbent stepping and cycling will expand and improve the understanding of locomotor adaptation processes beyond just perturbed gait. Using a perturbed recumbent stepping protocol, we hypothesized motor errors and anterior cingulate activity would decrease with time, and perturbation timing would influence electrocortical elicitation. Young adults (n = 17) completed four 10-minute arms and legs stepping tasks, with perturbations applied at every left or right leg extension-onset or mid-extension. A random no-perturbation "catch" stride occurred in every five perturbed strides. We instructed subjects to follow a pacing cue and to step smoothly, and we quantified temporal and spatial motor errors. We used high-density electroencephalography to estimate sources of electrocortical fluctuations shared among >70% of subjects. Temporal and spatial errors did not decrease from early to late for either perturbed or catch strides. Interestingly, spatial errors post-perturbation did not return to pre-perturbation levels, suggesting use-dependent learning occurred. Theta (3-8 Hz) synchronization in the anterior cingulate cortex and left and right supplementary motor areas (SMA) emerged near the perturbation event, and extension-onset perturbations elicited greater theta-band power than mid-extension perturbations. Even though motor errors did not adapt, anterior cingulate theta synchronization decreased from early to late perturbed strides, but only during the right-side tasks. Additionally, SMA mainly demonstrated specialized, not contralateral, lateralization. Overall, seated locomotor perturbations produced differential theta-band responses in the anterior cingulate and SMAs, suggesting that tuning perturbation parameters, e.g., timing, can potentially modify electrocortical responses.

Task Feedback Processing Differs Between Young and Older Adults in Visuomotor Rotation Learning Despite Similar Initial Adaptation and Savings

Ageing has been suggested to affect sensorimotor adaptation by impairing explicit strategy use. Here we recorded electrophysiological (EEG) responses during visuomotor rotation in both young (n = 24) and older adults (n = 25), to investigate the neural processes that underpin putative age-related effects on adaptation. We measured the feedback related negativity (FRN) and the P3 in response to task-feedback, as electrophysiological markers of task error processing and outcome evaluation. The two age groups adapted similarly and showed comparable after effects and savings when re-exposed to the same perturbation several days after the initial session. Older adults, however, had less distinct EEG responses (i.e., reduced FRN amplitudes) to negative and positive task feedback. The P3 did not differ between age groups. Both young and older adults also showed a sustained late positivity following task feedback. Measured at the frontal electrode Fz, this sustained activity was negatively associated with both the amount of voluntary disengagement of explicit strategy and savings. In conclusion, despite preserved task performance, we find clear differences in neural responses to errors in older people, which suggests that there is a fundamental decline in this aspect of sensorimotor brain function with age.

Reinforcement regulates timing variability in thalamus

Learning reduces variability but variability can facilitate learning. This paradoxical relationship has made it challenging to tease apart sources of variability that degrade performance from those that improve it. We tackled this question in a context-dependent timing task requiring humans and monkeys to flexibly produce different time intervals with different effectors. We identified two opposing factors contributing to timing variability: slow memory fluctuation that degrades performance and reward-dependent exploratory behavior that improves performance. Signatures of these opposing factors were evident across populations of neurons in the dorsomedial frontal cortex (DMFC), DMFC-projecting neurons in the ventrolateral thalamus, and putative target of DMFC in the caudate. However, only in the thalamus were the performance-optimizing regulation of variability aligned to the slow performance-degrading memory fluctuations. These findings reveal how variability caused by exploratory behavior might help to mitigate other undesirable sources of variability and highlight a potential role for thalamocortical projections in this process.

Brain Activity Reveals Multiple Motor-Learning Mechanisms in a Real-World Task

Many recent studies found signatures of motor learning in neural beta oscillations (13-30 Hz), and specifically in the post-movement beta rebound (PMBR). All these studies were in controlled laboratory-tasks in which the task designed to induce the studied learning mechanism. Interestingly, these studies reported opposing dynamics of the PMBR magnitude over learning for the error-based and reward-based tasks (increase vs. decrease, respectively). Here, we explored the PMBR dynamics during real-world motor-skill-learning in a billiards task using mobile-brain-imaging. Our EEG recordings highlight the opposing dynamics of PMBR magnitudes (increase vs. decrease) between different subjects performing the same task. The groups of subjects, defined by their neural dynamics, also showed behavioral differences expected for different learning mechanisms. Our results suggest that when faced with the complexity of the real-world different subjects might use different learning mechanisms for the same complex task. We speculate that all subjects combine multi-modal mechanisms of learning, but different subjects have different predominant learning mechanisms.

Punishment Feedback Impairs Memory and Changes Cortical Feedback-Related Potentials During Motor Learning

Reward and punishment have demonstrated dissociable effects on motor learning and memory, which suggests that these reinforcers are differently processed by the brain. To test this possibility, we use electroencephalography to record cortical neural activity after the presentation of reward and punishment feedback during a visuomotor rotation task. Participants were randomly placed into Reward, Punishment, or Control groups and performed the task under different conditions to assess the adaptation (learning) and retention (memory) of the motor task. These conditions featured an incongruent position between the cursor and the target, with the cursor trajectory, rotated 30° counterclockwise, requiring the participant to adapt their movement to hit the target. Feedback based on error magnitude was provided during the Adaptation condition in the form of a positive number (Reward) or negative number (Punishment), each representing a monetary gain or loss, respectively. No reinforcement or visual feedback was provided during the No Vision condition (retention). Performance error and event-related potentials (ERPs) time-locked to feedback presentation were calculated for each participant during both conditions. Punishment feedback reduced performance error and promoted faster learning during the Adaptation condition. In contrast, punishment feedback increased performance error during the No Vision condition compared to Control and Reward groups, which suggests a diminished motor memory. Moreover, the Punishment group showed a significant decrease in the amplitude of ERPs during the No Vision condition compared to the Adaptation condition. The amplitude of ERPs did not change in the other two groups. These results suggest that punishment feedback impairs motor retention by altering the neural processing involved in memory encoding. This study provides a neurophysiological underpinning for the dissociative effects of punishment feedback on motor learning.

EEG correlates of physical effort and reward processing during reinforcement learning

Effort-based decision making is often modeled using subjective value, a function of reward discounted by effort. We asked whether EEG event-related potential (ERP) correlates of reward processing are also modulated by physical effort. Human participants performed a task in which they were required to accurately produce target levels of muscle activation to receive rewards. Quadriceps muscle activation was recorded with electromyography (EMG) during isometric knee extension. On a given trial, the target muscle activation required either low or high effort. The effort was determined probabilistically according to a binary choice, such that the responses were associated with 20% and 80% probability of high effort. This contingency could only be learned through experience, and it reversed periodically. Binary reinforcement feedback depended on accurately producing the target muscle activity. Participants adaptively avoided effort by switching responses more frequently after choices that resulted in hard effort. Feedback after participants' choices that revealed the resulting effort requirement did not elicit modulation of the feedback-related negativity/reward positivity (FRN/RP). However, the neural response to reinforcement outcome after effort production was increased by preceding physical effort. Source decomposition revealed separable early and late positive deflections contributing to the ERP. The main effect of reward outcome, characteristic of the FRN/RP, loaded onto the earlier component, whereas the reward × effort interaction was observed only in the later positivity, which resembled the P300. Thus, retrospective effort modulates reward processing. This may explain paradoxical behavioral findings whereby rewards requiring more effort to obtain can become more powerful reinforcers.NEW & NOTEWORTHY Choices probabilistically determined the physical effort requirements for a subsequent task, and reward depended on task performance. Feedback revealing whether choices resulted in easy or hard effort did not elicit reinforcement learning signals. However, the neural responses to reinforcement were modulated by preceding effort. Thus, effort itself was not treated as loss or punishment, but it affected the responses to subsequent reinforcement outcomes. This may explain how effort can enhance the motivational effect of reward.

Visual Feedback Modulates Aftereffects and Electrophysiological Markers of Prism Adaptation

Prism adaptation (PA) is both a model for visuomotor learning and a promising treatment for visuospatial neglect after stroke. The task involves reaching for targets while prism glasses horizontally displace the visual field. Adaptation is hypothesized to occur through two processes: strategic recalibration, a rapid self-correction of pointing errors; and spatial realignment, a more gradual adjustment of visuomotor reference frames that produce prism aftereffects (i.e., reaching errors upon glasses removal in the direction opposite to the visual shift). While aftereffects can ameliorate neglect, not all patients respond to PA, and the neural mechanisms underlying successful adaptation are unclear. We investigated the feedback-related negativity (FRN) and the P300 event-related potential (ERP) components as candidate markers of strategic recalibration and spatial realignment, respectively. Healthy young adults wore prism glasses and performed memory-guided reaching toward vertical-line targets. ERPs were recorded in response to three different between-subject error feedback conditions at screen-touch: view of hand and target (Experiment 1), view of hand only (Experiment 2), or view of lines to mark target and hand position (view of hand occluded; Experiment 3). Conditions involving a direct view of the hand-produced stronger aftereffects than indirect hand feedback, and also evoked a P300 that decreased in amplitude as adaptation proceeded. Conversely, the FRN was only seen in conditions involving target feedback, even when aftereffects were smaller. Since conditions producing stronger aftereffects were associated with a phase-sensitive P300, this component may index a “context-updating” realignment process critical for strong aftereffects, whereas the FRN may reflect an error monitoring process related to strategic recalibration.

Effect of Sensory Loss on Improvements of Upper-Limb Paralysis Through Robot-Assisted Training: A Preliminary Case Series Study

Sensory disorder is a factor preventing recovery from motor paralysis after stroke. Although several robot-assisted exercises for the hemiplegic upper limb of stroke patients have been proposed, few studies have examined improvement in function in stroke patients with sensory disorder using robot-assisted training. In this study, the efficacies of robot training for the hemiplegic upper limb of three stroke patients with complete sensory loss were compared with those of 19 patients without complete sensory loss. Robot training to assist reach motion was performed in 10 sessions over a 2-week period for 5 days per week at 1 h per day. Before and after the training, the total Fugl–Meyer Assessment score excluding coordination and tendon reflex (FMA-total) and the FMA shoulder and elbow score excluding tendon reflex (FMA-S/E) were evaluated. Reach and path errors (RE and PE) during the reach motion were also evaluated by the arm-training robot. In most cases, both the FMA-total and the FMA-S/E scores improved. Cases with complete sensory loss showed worse RE and PE scores. Our results suggest that motor paralysis is improved by robot training. However, improvement may be varied according to the presence or absence of somatic sensory feedback.

The gradient of the reinforcement landscape influences sensorimotor learning

Consideration of previous successes and failures is essential to mastering a motor skill. Much of what we know about how humans and animals learn from such reinforcement feedback comes from experiments that involve sampling from a small number of discrete actions. Yet, it is less understood how we learn through reinforcement feedback when sampling from a continuous set of possible actions. Navigating a continuous set of possible actions likely requires using gradient information to maximize success. Here we addressed how humans adapt the aim of their hand when experiencing reinforcement feedback that was associated with a continuous set of possible actions. Specifically, we manipulated the change in the probability of reward given a change in motor action-the reinforcement gradient-to study its influence on learning. We found that participants learned faster when exposed to a steep gradient compared to a shallow gradient. Further, when initially positioned between a steep and a shallow gradient that rose in opposite directions, participants were more likely to ascend the steep gradient. We introduce a model that captures our results and several features of motor learning. Taken together, our work suggests that the sensorimotor system relies on temporally recent and spatially local gradient information to drive learning.