Pitfalls in quantifying exploration in reward-based motor learning and how to avoid them

Punishment Leads to Greater Sensorimotor Learning But Less Movement Variability Compared to Reward

When a musician practices a new song, hitting a correct note sounds pleasant while striking an incorrect note sounds unpleasant. Such reward and punishment feedback has been shown to differentially influence the ability to learn a new motor skill. Recent work has suggested that punishment leads to greater movement variability, which causes greater exploration and faster learning. To further test this idea, we collected 102 participants over two experiments. Unlike previous work, in Experiment 1 we found that punishment did not lead to faster learning compared to reward (n = 68), but did lead to a greater extent of learning. Surprisingly, we also found evidence to suggest that punishment led to less movement variability, which was related to the extent of learning. We then designed a second experiment that did not involve adaptation, allowing us to further isolate the influence of punishment feedback on movement variability. In Experiment 2, we again found that punishment led to significantly less movement variability compared to reward (n = 34). Collectively our results suggest that punishment feedback leads to less movement variability. Future work should investigate whether punishment feedback leads to a greater knowledge of movement variability and or increases the sensitivity of updating motor actions.

Reinforcement Learning during Locomotion

When learning a new motor skill, people often must use trial and error to discover which movement is best. In the reinforcement learning framework, this concept is known as exploration and has been linked to increased movement variability in motor tasks. For locomotor tasks, however, increased variability decreases upright stability. As such, exploration during gait may jeopardize balance and safety, making reinforcement learning less effective. Therefore, we set out to determine if humans could acquire and retain a novel locomotor pattern using reinforcement learning alone. Young healthy male and female participants walked on a treadmill and were provided with binary reward feedback (indicated by a green checkmark on the screen) that was tied to a fixed monetary bonus, to learn a novel stepping pattern. We also recruited a comparison group who walked with the same novel stepping pattern but did so by correcting for target error, induced by providing real-time veridical visual feedback of steps and a target. In two experiments, we compared learning, motor variability, and two forms of motor memories between the groups. We found that individuals in the binary reward group did, in fact, acquire the new walking pattern by exploring (increasing motor variability). Additionally, while reinforcement learning did not increase implicit motor memories, it resulted in more accurate explicit motor memories compared with the target error group. Overall, these results demonstrate that humans can acquire new walking patterns with reinforcement learning and retain much of the learning over 24 h.

Reinforcement-based processes actively regulate motor exploration along redundant solution manifolds

From a baby's babbling to a songbird practising a new tune, exploration is critical to motor learning. A hallmark of exploration is the emergence of random walk behaviour along solution manifolds, where successive motor actions are not independent but rather become serially dependent. Such exploratory random walk behaviour is ubiquitous across species' neural firing, gait patterns and reaching behaviour. The past work has suggested that exploratory random walk behaviour arises from an accumulation of movement variability and a lack of error-based corrections. Here, we test a fundamentally different idea-that reinforcement-based processes regulate random walk behaviour to promote continual motor exploration to maximize success. Across three human reaching experiments, we manipulated the size of both the visually displayed target and an unseen reward zone, as well as the probability of reinforcement feedback. Our empirical and modelling results parsimoniously support the notion that exploratory random walk behaviour emerges by utilizing knowledge of movement variability to update intended reach aim towards recently reinforced motor actions. This mechanism leads to active and continuous exploration of the solution manifold, currently thought by prominent theories to arise passively. The ability to continually explore muscle, joint and task redundant solution manifolds is beneficial while acting in uncertain environments, during motor development or when recovering from a neurological disorder to discover and learn new motor actions.

Implicit reward-based motor learning

Binary feedback, providing information solely about task success or failure, can be sufficient to drive motor learning. While binary feedback can induce explicit adjustments in movement strategy, it remains unclear if this type of feedback also induce implicit learning. We examined this question in a center-out reaching task by gradually moving an invisible reward zone away from a visual target to a final rotation of 7.5° or 25° in a between-group design. Participants received binary feedback, indicating if the movement intersected the reward zone. By the end of the training, both groups modified their reach angle by about 95% of the rotation. We quantified implicit learning by measuring performance in a subsequent no-feedback aftereffect phase, in which participants were told to forgo any adopted movement strategies and reach directly to the visual target. The results showed a small, but robust (2-3°) aftereffect in both groups, highlighting that binary feedback elicits implicit learning. Notably, for both groups, reaches to two flanking generalization targets were biased in the same direction as the aftereffect. This pattern is at odds with the hypothesis that implicit learning is a form of use-dependent learning. Rather, the results suggest that binary feedback can be sufficient to recalibrate a sensorimotor map.

Failure induces task-irrelevant exploration during a stencil task

During reward-based motor tasks, performance failure leads to an increase in movement variability along task-relevant dimensions. These increases in movement variability are indicative of exploratory behaviour in search of a better, more successful motor action. It is unclear whether failure also induces exploration along task-irrelevant dimensions that do not influence performance. In this study, we ask whether participants would explore the task-irrelevant dimension while they performed a stencil task. With a stylus, participants applied downward, normal force that influenced whether they received reward (task-relevant) as they simultaneously made erasing-like movement patterns along the tablet that did not influence performance (task-irrelevant). In this task, the movement pattern was analyzed as the distribution of movement directions within a movement. The results showed significant exploration of task-relevant force and task-irrelevant movement patterns. We conclude that failure can induce additional movement variability along a task-irrelevant dimension.

Parental praise and children's exploration: a virtual reality experiment

When children practice a new skill and fail, it is critical for them to explore new strategies to succeed. How can parents encourage children's exploration? Bridging insights from developmental psychology and the neuroscience of motor control, we examined the effects of parental praise on children's motor exploration. We theorize that modest praise can spark exploration. Unlike inflated praise, modest praise acknowledges children's performance, without setting a high standard for future performance. This may be reassuring to children with lower levels of self-esteem, who often doubt their ability. We conducted a novel virtual-reality experiment. Children (N = 202, ages 8-12) reported self-esteem and performed a virtual-reality 3D trajectory-matching task, with success/failure feedback after each trial. Children received modest praise ("You did well!"), inflated praise ("You did incredibly well!"), or no praise from their parent. We measured motor exploration as children's tendency to vary their movements following failure. Relative to no praise, modest praise-unlike inflated praise-encouraged exploration in children with lower levels of self-esteem. By contrast, modest praise discouraged exploration in children with higher levels of self-esteem. Effects were small yet robust. This experiment demonstrates that modest praise can spark exploration in children with lower levels of self-esteem.