Predictive Movements and Human Reinforcement Learning of Sequential Action

A Critical Period for Robust Curriculum-Based Deep Reinforcement Learning of Sequential Action in a Robot Arm

Many everyday activities are sequential in nature. That is, they can be seen as a sequence of subactions and sometimes subgoals. In the motor execution of sequential action, context effects are observed in which later subactions modulate the execution of earlier subactions (e.g., reaching for an overturned mug, people will optimize their grasp to achieve a comfortable end state). A trajectory (movement) adaptation of an often‐used paradigm in the study of sequential action, the serial response time task, showed several context effects of which centering behavior is of special interest. Centering behavior refers to the tendency (or strategy) of subjects to move their arm or mouse cursor to a position equidistant to all stimuli in the absence of predictive information, thereby reducing movement time to all possible targets. In the current study, we investigated sequential action in a virtual robotic agent trained using proximal policy optimization, a state‐of‐the‐art deep reinforcement learning algorithm. The agent was trained to reach for appearing targets, similar to a serial response time task given to humans. We found that agents were more likely to develop centering behavior similar to human subjects after curricularized learning. In our curriculum, we first rewarded agents for reaching targets before introducing a penalty for energy expenditure. When the penalty was applied with no curriculum, many agents failed to learn the task due to a lack of action space exploration, resulting in high variability of agents' performance. Our findings suggest that in virtual agents, similar to infants, early energetic exploration can promote robust later learning. This may have the same effect as infants' curiosity‐based learning by which they shape their own curriculum. However, introducing new goals cannot wait too long, as there may be critical periods in development after which agents (as humans) cannot flexibly learn to incorporate new objectives. These lessons are making their way into machine learning and offer exciting new avenues for studying both human and machine learning of sequential action.

In a sequential reaching task, a robot arm trained using deep reinforcement learning optimizes its behavior in a similar manner to humans. This so‐called centering behavior, however, only tends to develop when learning is performed in a curriculum in which the reward function becomes increasingly complex over time. Furthermore, the onset of the curriculum seems to have a sensitive period, with too early or too late onset leading to unstable or suboptimal performance.

The impact of implicit and explicit suggestions that 'there is nothing to learn' on implicit sequence learning

We can sometimes efficiently pick up statistical regularities in our environment in the absence of clear intentions or awareness, a process typically referred to as implicit sequence learning. In the current study, we tried to address the question whether suggesting participants that there is nothing to learn can impact this form of learning. If a priori predictions or intentions to learn are important in guiding implicit learning, we reasoned that suggesting participants that there is nothing to learn in a given context should hamper implicit learning. We introduced participants to random contexts that indicated that there was nothing to learn, either implicitly (i.e., by presenting blocks of random trials in "Experiment 1"), or explicitly (i.e., by explicitly instructing them in "Experiment 2"). Next, in a subsequent learning phase, participants performed an implicit sequence learning task. We found that these implicit or explicit suggestions that 'there was nothing to learn' did not influence the emergence of implicit knowledge in the subsequent learning phase. Although these findings seem consistent with simple associative or Hebbian learning accounts of implicit sequence learning (i.e., not steered by predictions), we discuss potential limitations that should inform future studies on the role of a priori predictions in implicit learning.