Dynamics systems vs. optimal control — a unifying view

Decoding Upper-Limb Movement Intention Through Adaptive Dynamic Movement Primitives: A Proof-of-Concept Study with a Shoulder-Elbow Exoskeleton

This work presents an intention decoding algorithm that can be used to control a 4 degrees-of-freedom shoulder-elbow exoskeleton in reaching tasks. The algorithm was designed to assist the movement of users with upper-limb impairments who can initiate the movement by themselves. It relies on the observation of the initial part of the user's movement through joint angle measures and aims to estimate in real-time the phase of the movement and predict the goal position of the hand in the reaching task. The algorithm is based on adaptive Dynamic Movement Primitives and Gaussian Mixture Models. The performance of the algorithm was verified in robot-assisted planar reaching movements performed by one healthy subject wearing the exoskeleton. Tests included movements of different amplitudes and orientations. Results showed that the algorithm could predict the hand's final position with an error lower than 5 cm after 0.25 s from the movement onset, and that the final position reached during the tests was on average less than 4 cm far from the target position. Finally, the effects of the assistance were observed in a reduction of the activation of the Biceps Brachii and of the time to execute the reaching tasks.

How the conception of control influences our understanding of actions

Wilful movement requires neural control. Commonly, neural computations are thought to generate motor commands that bring the musculoskeletal system - that is, the plant - from its current physical state into a desired physical state. The current state can be estimated from past motor commands and from sensory information. Modelling movement on the basis of this concept of plant control strives to explain behaviour by identifying the computational principles for control signals that can reproduce the observed features of movements. From an alternative perspective, movements emerge in a dynamically coupled agent-environment system from the pursuit of subjective perceptual goals. Modelling movement on the basis of this concept of perceptual control aims to identify the controlled percepts and their coupling rules that can give rise to the observed characteristics of behaviour. In this Perspective, we discuss a broad spectrum of approaches to modelling human motor control and their notions of control signals, internal models, handling of sensory feedback delays and learning. We focus on the influence that the plant control and the perceptual control perspective may have on decisions when modelling empirical data, which may in turn shape our understanding of actions.

Robot Programming from a Single Demonstration for High Precision Industrial Insertion

We propose a novel approach for robotic industrial insertion tasks using the Programming by Demonstration technique. Our method allows robots to learn a high-precision task by observing human demonstration once, without requiring any prior knowledge of the object. We introduce an Imitated-to-Finetuned approach that generates imitated approach trajectories by cloning the human hand's movements and then fine-tunes the goal position with a visual servoing approach. To identify features on the object used in visual servoing, we model object tracking as the moving object detection problem, separating each demonstration video frame into the moving foreground that includes the object and demonstrator's hand and the static background. Then a hand keypoints estimation function is used to remove the redundant features on the hand. The experiment shows that the proposed method can make robots learn precision industrial insertion tasks from a single human demonstration.

Dynamic branching in a neural network model for probabilistic prediction of sequences

An important function of the brain is to predict which stimulus is likely to occur based on the perceived cues. The present research studied the branching behavior of a computational network model of populations of excitatory and inhibitory neurons, both analytically and through simulations. Results show how synaptic efficacy, retroactive inhibition and short-term synaptic depression determine the dynamics of selection between different branches predicting sequences of stimuli of different probabilities. Further results show that changes in the probability of the different predictions depend on variations of neuronal gain. Such variations allow the network to optimize the probability of its predictions to changing probabilities of the sequences without changing synaptic efficacy.

Impedance Variation and Learning Strategies in Human-Robot Interaction

In this survey, various concepts and methodologies developed over the past two decades for varying and learning the impedance or admittance of robotic systems that physically interact with humans are explored. For this purpose, the assumptions and mathematical formulations for the online adjustment of impedance models and controllers for physical human-robot interaction (HRI) are categorized and compared. In this systematic review, studies on: 1) variation and 2) learning of appropriate impedance elements are taken into account. These strategies are classified and described in terms of their objectives, points of view (approaches), and signal requirements (including position, HRI force, and electromyography activity). Different methods involving linear/nonlinear analyses (e.g., optimal control design and nonlinear Lyapunov-based stability guarantee) and the Gaussian approximation algorithms (e.g., Gaussian mixture model-based and dynamic movement primitives-based strategies) are reviewed. Current challenges and research trends in physical HRI are finally discussed.

The Impact of Nonminimum-Phase Zeros on Human-in-the-Loop Control Systems

We present results from an experiment in which 33 human subjects interact with a dynamic system 40 times over a one-week period. The subjects are divided into three groups. For each interaction, a subject performs a command-following task, where the reference command is the same for all trials and all subjects. However, each group interacts with a different dynamic system, which is represented by a transfer function. The transfer functions have the same poles but different zeros. One has a minimum-phase zero , another has a nonminimum-phase zero , and the last has a slower (i.e., closer to the imaginary axis) nonminimum-phase zero z_sn ∈ (0,z_n) . The experimental results show that nonminimum-phase zeros tend to make dynamic systems more difficult for humans to learn to control. We use a subsystem identification algorithm to identify the control strategy that each subject uses on each trial. The identification results show that the identified feedforward controllers approximate the inverse dynamics of the system with which the subjects interact better on the last trial than on the first trial. However, the subjects interacting with the minimum-phase system are able to approximate the inverse dynamics in feedforward more accurately than the subjects interacting with the nonminimum-phase system. This observation suggests that nonminimum-phase zeros are an impediment to approximating inverse dynamics in feedforward. Finally, we provide evidence that humans rely on feedforward-step-like-control strategies with systems (e.g., nonminimum-phase systems) for which it is difficult to approximate the inverse dynamics in feedforward.

Safe Motion Planning and Learning for Unmanned Aerial Systems

To control unmanned aerial systems, we rarely have a perfect system model. Safe and aggressive planning is also challenging for nonlinear and under-actuated systems. Expert pilots, however, demonstrate maneuvers that are deemed at the edge of plane envelope. Inspired by biological systems, in this paper, we introduce a framework that leverages methods in the field of control theory and reinforcement learning to generate feasible, possibly aggressive, trajectories. For the control policies, Dynamic Movement Primitives (DMPs) imitate pilot-induced primitives, and DMPs are combined in parallel to generate trajectories to reach original or different goal points. The stability properties of DMPs and their overall systems are analyzed using contraction theory. For reinforcement learning, Policy Improvement with Path Integrals (PI2) was used for the maneuvers. The results in this paper show that PI2 updated policies are a feasible and parallel combination of different updated primitives transfer the learning in the contraction regions. Our proposed methodology can be used to imitate, reshape, and improve feasible, possibly aggressive, maneuvers. In addition, we can exploit trajectories generated by optimization methods, such as Model Predictive Control (MPC), and a library of maneuvers can be instantly generated. For application, 3-DOF (degrees of freedom) Helicopter and 2D-UAV (unmanned aerial vehicle) models are utilized to demonstrate the main results.

Neuronal Sequence Models for Bayesian Online Inference

Various imaging and electrophysiological studies in a number of different species and brain regions have revealed that neuronal dynamics associated with diverse behavioral patterns and cognitive tasks take on a sequence-like structure, even when encoding stationary concepts. These neuronal sequences are characterized by robust and reproducible spatiotemporal activation patterns. This suggests that the role of neuronal sequences may be much more fundamental for brain function than is commonly believed. Furthermore, the idea that the brain is not simply a passive observer but an active predictor of its sensory input, is supported by an enormous amount of evidence in fields as diverse as human ethology and physiology, besides neuroscience. Hence, a central aspect of this review is to illustrate how neuronal sequences can be understood as critical for probabilistic predictive information processing, and what dynamical principles can be used as generators of neuronal sequences. Moreover, since different lines of evidence from neuroscience and computational modeling suggest that the brain is organized in a functional hierarchy of time scales, we will also review how models based on sequence-generating principles can be embedded in such a hierarchy, to form a generative model for recognition and prediction of sensory input. We shortly introduce the Bayesian brain hypothesis as a prominent mathematical description of how online, i.e., fast, recognition, and predictions may be computed by the brain. Finally, we briefly discuss some recent advances in machine learning, where spatiotemporally structured methods (akin to neuronal sequences) and hierarchical networks have independently been developed for a wide range of tasks. We conclude that the investigation of specific dynamical and structural principles of sequential brain activity not only helps us understand how the brain processes information and generates predictions, but also informs us about neuroscientific principles potentially useful for designing more efficient artificial neuronal networks for machine learning tasks.

Motion Adaptation Based on Learning the Manifold of Task and Dynamic Movement Primitive Parameters

Dynamic movement primitives (DMP) are motion building blocks suitable for real-world tasks. We suggest a methodology for learning the manifold of task and DMP parameters, which facilitates runtime adaptation to changes in task requirements while ensuring predictable and robust performance. For efficient learning, the parameter space is analyzed using principal component analysis and locally linear embedding. Two manifold learning methods: kernel estimation and deep neural networks, are investigated for a ball throwing task in simulation and in a physical environment. Low runtime estimation errors are obtained for both learning methods, with an advantage to kernel estimation when data sets are small.

Human control of complex objects: Towards more dexterous robots

Manipulation of objects with underactuated dynamics remains a challenge for robots. In contrast, humans excel at 'tool use' and more insight into human control strategies may inform robotic control architectures. We examined human control of objects that exhibit complex - underactuated, nonlinear, and potentially chaotic dynamics, such as transporting a cup of coffee. Simple control strategies appropriate for unconstrained movements, such as maximizing smoothness, fail as interaction forces have to be compensated or preempted. However, predictive control based on internal models appears daunting when the objects have nonlinear and unpredictable dynamics. We hypothesized that humans learn strategies that make these interactions predictable. Using a virtual environment subjects interacted with a virtual cup and rolling ball using a robotic visual and haptic interface. Two different metrics quantified predictability: stability or contraction, and mutual information between controller and object. In point-to-point displacements subjects exploited the contracting regions of the object dynamics to safely navigate perturbations. Control contraction metrics showed that subjects used a controller that exponentially stabilized trajectories. During continuous cup-and-ball displacements subjects developed predictable solutions sacrificing smoothness and energy efficiency. These results may stimulate control strategies for dexterous robotic manipulators and human-robot interaction.

Robustness in Human Manipulation of Dynamically Complex Objects through Control Contraction Metrics

Control and manipulation of objects with underactuated dynamics remains a challenge for robots. Due to their typically nonlinear dynamics, it is computationally taxing to implement model-based planning and control techniques. Yet humans can skillfully manipulate such objects, seemingly with ease. More insight into human control strategies may inform how to enhance control strategies in robots. This study examined human control of objects that exhibit complex - underactuated and nonlinear - dynamics. We hypothesized that humans seek to make their trajectories exponentially stable to achieve robustness in the face of external perturbations. A stable trajectory is also robust to the high levels of noise in the human neuromotor system. Motivated by the task of carrying a cup of coffee, a virtual implementation of transporting a cart-pendulum system was developed. Subjects interacted with the virtual system via a robotic manipulandum that provided a haptic and visual interface. Human subjects were instructed to transport this simplified system to a target position as fast as possible without 'spilling coffee', while accommodating different visible perturbations that could be anticipated. To test the hypothesis of exponential convergence, tools from the framework of control contraction metrics were leveraged to analyze human trajectories. Results showed that with practice the trajectories indeed became exponentially stable, selectively around the perturbation. While these findings are agnostic about the involvement of feedback and feedforward control, they do support the hypothesis that humans learn to make trajectories stable, consistent with achieving predictability.

Bridging Dynamical Systems and Optimal Trajectory Approaches to Speech Motor Control With Dynamic Movement Primitives

Current models of speech motor control rely on either trajectory-based control (DIVA, GEPPETO, ACT) or a dynamical systems approach based on feedback control (Task Dynamics, FACTS). While both approaches have provided insights into the speech motor system, it is difficult to connect these findings across models given the distinct theoretical and computational bases of the two approaches. We propose a new extension of the most widely used dynamical systems approach, Task Dynamics, that incorporates many of the strengths of trajectory-based approaches, providing a way to bridge the theoretical divide between what have been two separate approaches to understanding speech motor control. The Task Dynamics (TD) model posits that speech gestures are governed by point attractor dynamics consistent with a critically damped harmonic oscillator. Kinematic trajectories associated with such gestures should therefore be consistent with a second-order dynamical system, possibly modified by blending with temporally overlapping gestures or altering oscillator parameters. This account of observed kinematics is powerful and theoretically appealing, but may be insufficient to account for deviations from predicted kinematics-i.e., changes produced in response to some external perturbations to the jaw, changes in control during acquisition and development, or effects of word/syllable frequency. Optimization, such as would be needed to minimize articulatory effort, is also incompatible with the current TD model, though the idea that the speech production systems economizes effort has a long history and, importantly, also plays a critical role in current theories of domain-general human motor control. To address these issues, we use Dynamic Movement Primitives (DMPs) to expand a dynamical systems framework for speech motor control to allow modification of kinematic trajectories by incorporating a simple, learnable forcing term into existing point attractor dynamics. We show that integration of DMPs with task-based point-attractor dynamics enhances the potential explanatory power of TD in a number of critical ways, including the ability to account for external forces in planning and optimizing both kinematic and dynamic movement costs. At the same time, this approach preserves the successes of Task Dynamics in handling multi-gesture planning and coordination.

Small-variance asymptotics for non-parametric online robot learning

Small-variance asymptotics is emerging as a useful technique for inference in large-scale Bayesian non-parametric mixture models. This paper analyzes the online learning of robot manipulation tasks with Bayesian non-parametric mixture models under small-variance asymptotics. The analysis yields a scalable online sequence clustering (SOSC) algorithm that is non-parametric in the number of clusters and the subspace dimension of each cluster. SOSC groups the new datapoint in low-dimensional subspaces by online inference in a non-parametric mixture of probabilistic principal component analyzers (MPPCA) based on a Dirichlet process, and captures the state transition and state duration information online in a hidden semi-Markov model (HSMM) based on a hierarchical Dirichlet process. A task-parameterized formulation of our approach autonomously adapts the model to changing environmental situations during manipulation. We apply the algorithm in a teleoperation setting to recognize the intention of the operator and remotely adjust the movement of the robot using the learned model. The generative model is used to synthesize both time-independent and time-dependent behaviors by relying on the principles of shared and autonomous control. Experiments with the Baxter robot yield parsimonious clusters that adapt online with new demonstrations and assist the operator in performing remote manipulation tasks.

Lower limb joint angle variability and dimensionality are different in stairmill climbing and treadmill walking

The present study tested if the quadratic relationship which exists between stepping frequency and gait dynamics in walking can be generalized to stairmill climbing. To accomplish this, we investigated the joint angle dynamics and variability during continuous stairmill climbing at stepping frequencies both above and below the preferred stepping frequency (PSF). Nine subjects performed stairmill climbing at 80, 90, 100, 110 and 120% PSF and treadmill walking at preferred walking speed during which sagittal hip, knee and ankle angles were extracted. Joint angle dynamics were quantified by the largest Lyapunov exponent (LyE) and correlation dimension (CoD). Joint angle variability was estimated by the mean ensemble standard deviation (meanSD). MeanSD and CoD for all joints were significantly higher during stairmill climbing but there were no task differences in LyE. Changes in stepping frequency had only limited effect on joint angle variability and did not affect joint angle dynamics. Thus, we concluded that the quadratic relationship between stepping frequency and gait dynamics observed in walking is not present in stairmill climbing based on the investigated parameters.

Fast social-like learning of complex behaviors based on motor motifs

Social learning is widely observed in many species. Less experienced agents copy successful behaviors exhibited by more experienced individuals. Nevertheless, the dynamical mechanisms behind this process remain largely unknown. Here we assume that a complex behavior can be decomposed into a sequence of n motor motifs. Then a neural network capable of activating motor motifs in a given sequence can drive an agent. To account for (n-1)! possible sequences of motifs in a neural network, we employ the winnerless competition approach. We then consider a teacher-learner situation: one agent exhibits a complex movement, while another one aims at mimicking the teacher's behavior. Despite the huge variety of possible motif sequences we show that the learner, equipped with the provided learning model, can rewire "on the fly" its synaptic couplings in no more than (n-1) learning cycles and converge exponentially to the durations of the teacher's motifs. We validate the learning model on mobile robots. Experimental results show that the learner is indeed capable of copying the teacher's behavior composed of six motor motifs in a few learning cycles. The reported mechanism of learning is general and can be used for replicating different functions, including, for example, sound patterns or speech.

Shape-based coordination in locomotion control

Highly articulated systems are capable of executing a variety of behaviors by coordinating their many internal degrees of freedom to help them move more effectively in complex terrains. However, this inherent variety poses significant challenges that have been the subject of a great deal of previous work: What are the most effective or most efficient methods for achieving the intrinsic coordination necessary to produce desired global objectives? This work takes these questions one step further, asking how different levels of coordination, which we quantify in terms of kinematic coupling, affect articulated locomotion in environments with different degrees of underlying structure. We introduce shape functions as the analytical basis for specifying kinematic coupling relationships that constrain the relative motion among the internal degrees of freedom for a given system during its nominal locomotion. Furthermore, we show how shape functions are used to derive shape-based controllers (SBCs) that manage the compliant interaction between articulated bodies and the environment while explicitly preserving the inter-joint coupling defined by shape functions. Initial experimental evidence provides a comparison of the benefits of different levels of coordination for two separate platforms in environments with different degrees of inherent structure. The experimental results show that decentralized implementations, where there is relatively little inter-joint coupling, perform well across a spectrum of different terrains but that there are potential benefits to higher degrees of coupling in structured terrains. We discuss how this observation has implications related to future planning and control approaches that actively “tune” their underlying structure by dynamically varying the assumed level of coupling as a function of task specification and local environmental conditions.

The Roles of Feedback and Feedforward as Humans Learn to Control Unknown Dynamic Systems

We present results from an experiment in which human subjects interact with an unknown dynamic system 40 times during a two-week period. During each interaction, subjects are asked to perform a command-following (i.e., pursuit tracking) task. Each subject's performance at that task improves from the first trial to the last trial. For each trial, we use subsystem identification to estimate each subject's feedforward (or anticipatory) control, feedback (or reactive) control, and feedback time delay. Over the 40 trials, the magnitudes of the identified feedback controllers and the identified feedback time delays do not change significantly. In contrast, the identified feedforward controllers do change significantly. By the last trial, the average identified feedforward controller approximates the inverse of the dynamic system. This observation provides evidence that a fundamental component of human learning is updating the anticipatory control until it models the inverse dynamics.

Whole-Body Movements in Long-Term Weightlessness: Hierarchies of the Controlled Variables Are Gravity-Dependent

Switching between contexts affects the mechanisms underlying motion planning, in particular it may entail reranking the variables to be controlled in defining the motor solutions. Three astronauts performed multiple sessions of whole-body pointing, in normogravity before launch, in prolonged weightlessness onboard the International Space Station, and after return. The effect of gravity context on kinematic and dynamic components was evaluated. Hand trajectory was gravity independent; center-of-mass excursion was highly variable within and between subjects. The body-environment effort exchange, expressed as inertial ankle momentum, was systematically lower in weightlessness than in normogravity. After return on Earth, the system underwent a rapid 1-week readaptation. The study indicates that minimizing the control effort is given greater weight when optimizing the motor plan in weightlessness compared to normogravity: the hierarchies of the controlled variables are gravity dependent.

Speed adaptation for self-improvement of skills learned from user demonstrations

The paper addresses the problem of speed adaptation of movements subject to environmental constraints. Our approach relies on a novel formulation of velocity profiles as an extension of dynamic movement primitives (DMP). The framework allows for compact representation of non-uniformly accelerated motion as well as simple modulation of the movement parameters. In the paper, we evaluate two model free methods by which optimal parameters can be obtained: iterative learning control (ILC) and policy search based reinforcement learning (RL). The applicability of each method is discussed and evaluated on two distinct cases, which are hard to model using standard techniques. The first deals with hard contacts with the environment while the second process involves liquid dynamics. We find ILC to be very efficient in cases where task parameters can be easily described with an error function. On the other hand, RL has stronger convergence properties and can therefore provide a solution in the general case.

An online gait generator for quadruped walking using motor primitives

This article presents implementation of an online gait generator on a quadruped robot. Firstly, the design of a quadruped robot is presented. The robot contains four leg modules each of which is constructed by a 2 degrees of freedom (2-DOF) five-bar parallel linkage mechanism. Together with other two rotational DOF, the leg module is able to perform 4-DOF movement. The parallel mechanism of the robot allows all the servos attached on the body frame, so that the leg mass is decreased and motor load can be balanced. Secondly, an online gait generator based on dynamic movement primitives for the walking control is presented. Dynamic movement primitives provide an approach to generate periodic trajectories and they can be modulated in real time, which makes the online adjustment of walking gaits possible. This gait controller is tested by the quadruped robot in regulating walking speed, switching between forward\backward movements and steering. The controller is easy to apply, expand and is quite effective on phase coordination and online trajectory modulation. Results of simulated experiments are presented.

Robotics and neuroscience

In the attempt to build adaptive and intelligent machines, roboticists have looked at neuroscience for more than half a century as a source of inspiration for perception and control. More recently, neuroscientists have resorted to robots for testing hypotheses and validating models of biological nervous systems. Here, we give an overview of the work at the intersection of robotics and neuroscience and highlight the most promising approaches and areas where interactions between the two fields have generated significant new insights. We articulate the work in three sections, invertebrate, vertebrate and primate neuroscience. We argue that robots generate valuable insight into the function of nervous systems, which is intimately linked to behaviour and embodiment, and that brain-inspired algorithms and devices give robots life-like capabilities.

The role of prediction and outcomes in adaptive cognitive control

Humans adaptively perform actions to achieve their goals. This flexible behaviour requires two core abilities: the ability to anticipate the outcomes of candidate actions and the ability to select and implement actions in a goal-directed manner. The ability to predict outcomes has been extensively researched in reinforcement learning paradigms, but this work has often focused on simple actions that are not embedded in hierarchical and sequential structures that are characteristic of goal-directed human behaviour. On the other hand, the ability to select actions in accordance with high-level task goals, particularly in the presence of alternative responses and salient distractors, has been widely researched in cognitive control paradigms. Cognitive control research, however, has often paid less attention to the role of action outcomes. The present review attempts to bridge these accounts by proposing an outcome-guided mechanism for selection of extended actions. Our proposal builds on constructs from the hierarchical reinforcement learning literature, which emphasises the concept of reaching and evaluating informative states, i.e., states that constitute subgoals in complex actions. We develop an account of the neural mechanisms that allow outcome-guided action selection to be achieved in a network that relies on projections from cortical areas to the basal ganglia and back-projections from the basal ganglia to the cortex. These cortico-basal ganglia-thalamo-cortical 'loops' allow convergence - and thus integration - of information from non-adjacent cortical areas (for example between sensory and motor representations). This integration is essential in action sequences, for which achieving an anticipated sensory state signals the successful completion of an action. We further describe how projection pathways within the basal ganglia allow selection between representations, which may pertain to movements, actions, or extended action plans. The model lastly envisages a role for hierarchical projections from the striatum to dopaminergic midbrain areas that enable more rostral frontal areas to bias the selection of inputs from more posterior frontal areas via their respective representations in the basal ganglia.

Is intraindividual variability different between unimanual and bimanual speed-accuracy movements?

Dominant (right) and nondominant (left) hand differences in motor performance variables were investigated during targeted rapid aiming unimanual and bimanual (to two separate targets) speed-accuracy tasks. The performance of the dominant and nondominant hands were compared during rapid targeted unimanual and bimanual reaching tasks (50 repetitions) in adult, neurologically intact, right-handed men (n = 20). As task difficulty increased from unimanual to bimanual tasks, reaction time increased and velocity (average and maximal) of performance decreased significantly. The effect of hand was significant only on average movement velocity and on intraindividual variability of accuracy (in both cases performance in the left hand was worse). Variability of movement path (accuracy) was less in the right hand during both unimanual and bimanual tasks.

Haptic feedback enhances rhythmic motor control by reducing variability, not improving convergence rate

Stability and performance during rhythmic motor behaviors such as locomotion are critical for survival across taxa: falling down would bode well for neither cheetah nor gazelle. Little is known about how haptic feedback, particularly during discrete events such as the heel-strike event during walking, enhances rhythmic behavior. To determine the effect of haptic cues on rhythmic motor performance, we investigated a virtual paddle juggling behavior, analogous to bouncing a table tennis ball on a paddle. Here, we show that a force impulse to the hand at the moment of ball-paddle collision categorically improves performance over visual feedback alone, not by regulating the rate of convergence to steady state (e.g., via higher gain feedback or modifying the steady-state hand motion), but rather by reducing cycle-to-cycle variability. This suggests that the timing and state cues afforded by haptic feedback decrease the nervous system's uncertainty of the state of the ball to enable more accurate control but that the feedback gain itself is unaltered. This decrease in variability leads to a substantial increase in the mean first passage time, a measure of the long-term metastability of a stochastic dynamical system. Rhythmic tasks such as locomotion and juggling involve intermittent contact with the environment (i.e., hybrid transitions), and the timing of such transitions is generally easy to sense via haptic feedback. This timing information may improve metastability, equating to less frequent falls or other failures depending on the task.

Reinforcement learning in robotics: A survey

Reinforcement learning offers to robotics a framework and set of tools for the design of sophisticated and hard-to-engineer behaviors. Conversely, the challenges of robotic problems provide both inspiration, impact, and validation for developments in reinforcement learning. The relationship between disciplines has sufficient promise to be likened to that between physics and mathematics. In this article, we attempt to strengthen the links between the two research communities by providing a survey of work in reinforcement learning for behavior generation in robots. We highlight both key challenges in robot reinforcement learning as well as notable successes. We discuss how contributions tamed the complexity of the domain and study the role of algorithms, representations, and prior knowledge in achieving these successes. As a result, a particular focus of our paper lies on the choice between model-based and model-free as well as between value-function-based and policy-search methods. By analyzing a simple problem in some detail we demonstrate how reinforcement learning approaches may be profitably applied, and we note throughout open questions and the tremendous potential for future research.

Dynamical Movement Primitives: Learning Attractor Models for Motor Behaviors

Nonlinear dynamical systems have been used in many disciplines to model complex behaviors, including biological motor control, robotics, perception, economics, traffic prediction, and neuroscience. While often the unexpected emergent behavior of nonlinear systems is the focus of investigations, it is of equal importance to create goal-directed behavior (e.g., stable locomotion from a system of coupled oscillators under perceptual guidance). Modeling goal-directed behavior with nonlinear systems is, however, rather difficult due to the parameter sensitivity of these systems, their complex phase transitions in response to subtle parameter changes, and the difficulty of analyzing and predicting their long-term behavior; intuition and time-consuming parameter tuning play a major role. This letter presents and reviews dynamical movement primitives, a line of research for modeling attractor behaviors of autonomous nonlinear dynamical systems with the help of statistical learning techniques. The essence of our approach is to start with a simple dynamical system, such as a set of linear differential equations, and transform those into a weakly nonlinear system with prescribed attractor dynamics by means of a learnable autonomous forcing term. Both point attractors and limit cycle attractors of almost arbitrary complexity can be generated. We explain the design principle of our approach and evaluate its properties in several example applications in motor control and robotics.

Learning to select and generalize striking movements in robot table tennis

Learning new motor tasks from physical interactions is an important goal for both robotics and machine learning. However, when moving beyond basic skills, most monolithic machine learning approaches fail to scale. For more complex skills, methods that are tailored for the domain of skill learning are needed. In this paper, we take the task of learning table tennis as an example and present a new framework that allows a robot to learn cooperative table tennis from physical interaction with a human. The robot first learns a set of elementary table tennis hitting movements from a human table tennis teacher by kinesthetic teach-in, which is compiled into a set of motor primitives represented by dynamical systems. The robot subsequently generalizes these movements to a wider range of situations using our mixture of motor primitives approach. The resulting policy enables the robot to select appropriate motor primitives as well as to generalize between them. Finally, the robot plays with a human table tennis partner and learns online to improve its behavior. We show that the resulting setup is capable of playing table tennis using an anthropomorphic robot arm.

The Role of Learning and Kinematic Features in Dexterous Manipulation: A Comparative Study with Two Robotic Hands

Dexterous movements performed by the human hand are by far more sophisticated than those achieved by current humanoid robotic hands and systems used to control them. This work aims at providing a contribution in order to overcome this gap by proposing a bio-inspired control architecture that captures two key elements underlying human dexterity. The first is the progressive development of skilful control, often starting from – or involving – cyclic movements, based on trial-and-error learning processes and central pattern generators. The second element is the exploitation of a particular kinematic features of the human hand, i.e. the thumb opposition. The architecture is tested with two simulated robotic hands having different kinematic features and engaged in rotating spheres, cylinders, and cubes of different sizes. The results support the feasibility of the proposed approach and show the potential of the model to allow a better understanding of the control mechanisms and kinematic principles underlying human dexterity and make them transferable to anthropomorphic robotic hands.

Strategy-dependent encoding of planned arm movements in the dorsal premotor cortex

The kinematic strategy encoded in motor cortical areas for classic straight-line reaching is remarkably simple and consistent across subjects, despite the complicated musculoskeletal dynamics that are involved. As tasks become more challenging, however, different conscious strategies may be used to improve perceived behavioral performance. We identified additional spatial information that appeared both in single neurons and in the population code of monkey dorsal premotor cortex when obstacles impeded direct reach paths. The neural correlate of movement planning varied between subjects in a manner consistent with the use of different strategies to optimize task completion. These distinct planning strategies were manifested in the timing and strength of the information contained in the neural population code.