Adaptive, fast walking in a biped robot under neuronal control and learning

The neuromechanics of animal locomotion: From biology to robotics and back

Robotics and neuroscience are sister disciplines that both aim to understand how agile, efficient, and robust locomotion can be achieved in autonomous agents. Robotics has already benefitted from neuromechanical principles discovered by investigating animals. These include the use of high-level commands to control low-level central pattern generator-like controllers, which, in turn, are informed by sensory feedback. Reciprocally, neuroscience has benefited from tools and intuitions in robotics to reveal how embodiment, physical interactions with the environment, and sensory feedback help sculpt animal behavior. We illustrate and discuss exemplar studies of this dialog between robotics and neuroscience. We also reveal how the increasing biorealism of simulations and robots is driving these two disciplines together, forging an integrative science of autonomous behavioral control with many exciting future opportunities.

Deep reinforcement learning for modeling human locomotion control in neuromechanical simulation

Modeling human motor control and predicting how humans will move in novel environments is a grand scientific challenge. Researchers in the fields of biomechanics and motor control have proposed and evaluated motor control models via neuromechanical simulations, which produce physically correct motions of a musculoskeletal model. Typically, researchers have developed control models that encode physiologically plausible motor control hypotheses and compared the resulting simulation behaviors to measurable human motion data. While such plausible control models were able to simulate and explain many basic locomotion behaviors (e.g. walking, running, and climbing stairs), modeling higher layer controls (e.g. processing environment cues, planning long-term motion strategies, and coordinating basic motor skills to navigate in dynamic and complex environments) remains a challenge. Recent advances in deep reinforcement learning lay a foundation for modeling these complex control processes and controlling a diverse repertoire of human movement; however, reinforcement learning has been rarely applied in neuromechanical simulation to model human control. In this paper, we review the current state of neuromechanical simulations, along with the fundamentals of reinforcement learning, as it applies to human locomotion. We also present a scientific competition and accompanying software platform, which we have organized to accelerate the use of reinforcement learning in neuromechanical simulations. This “Learn to Move” competition was an official competition at the NeurIPS conference from 2017 to 2019 and attracted over 1300 teams from around the world. Top teams adapted state-of-the-art deep reinforcement learning techniques and produced motions, such as quick turning and walk-to-stand transitions, that have not been demonstrated before in neuromechanical simulations without utilizing reference motion data. We close with a discussion of future opportunities at the intersection of human movement simulation and reinforcement learning and our plans to extend the Learn to Move competition to further facilitate interdisciplinary collaboration in modeling human motor control for biomechanics and rehabilitation research

Adaptive and Energy-Efficient Optimal Control in CPGs Through Tegotae-Based Feedback

To obtain biologically inspired robotic control, the architecture of central pattern generators (CPGs) has been extensively adopted to generate periodic patterns for locomotor control. This is attributed to the interesting properties of nonlinear oscillators. Although sensory feedback in CPGs is not necessary for the generation of patterns, it plays a central role in guaranteeing adaptivity to environmental conditions. Nonetheless, its inclusion significantly modifies the dynamics of the CPG architecture, which often leads to bifurcations. For instance, the force feedback can be exploited to derive information regarding the state of the system. In particular, the Tegotae approach can be adopted by coupling proprioceptive information with the state of the oscillation itself in the CPG model. This paper discusses this policy with respect to other types of feedback; it provides higher adaptivity and an optimal energy efficiency for reflex-like actuation. We believe this is the first attempt to analyse the optimal energy efficiency along with the adaptivity of the Tegotae approach.

Tegotae-Based Control Produces Adaptive Inter- and Intra-limb Coordination in Bipedal Walking

Despite the appealing concept of central pattern generator (CPG)-based control for bipedal walking robots, there is currently no systematic methodology for designing a CPG-based controller. To remedy this oversight, we attempted to apply the Tegotae approach, a Japanese concept describing how well a perceived reaction, i.e., sensory information, matches an expectation, i.e., an intended motor command, in designing localised controllers in the CPG-based bipedal walking model. To this end, we developed a Tegotae function that quantifies the Tegotae concept. This function allowed incorporating decentralised controllers into the proposed bipedal walking model systematically. We designed a two-dimensional bipedal walking model using Tegotae functions and subsequently implemented it in simulations to validate the proposed design scheme. We found that our model can walk on both flat and uneven terrains and confirmed that the application of the Tegotae functions in all joint controllers results in excellent adaptability to environmental changes.

Morphological Computation Increases From Lower- to Higher-Level of Biological Motor Control Hierarchy

Voluntary movements, like point-to-point or oscillatory human arm movements, are generated by the interaction of several structures. High-level neuronal circuits in the brain are responsible for planning and initiating a movement. Spinal circuits incorporate proprioceptive feedback to compensate for deviations from the desired movement. Muscle biochemistry and contraction dynamics generate movement driving forces and provide an immediate physical response to external forces, like a low-level decentralized controller. A simple central neuronal command like "initiate a movement" then recruits all these biological structures and processes leading to complex behavior, e.g., generate a stable oscillatory movement in resonance with an external spring-mass system. It has been discussed that the spinal feedback circuits, the biochemical processes, and the biomechanical muscle dynamics contribute to the movement generation, and, thus, take over some parts of the movement generation and stabilization which would otherwise have to be performed by the high-level controller. This contribution is termed morphological computation and can be quantified with information entropy-based approaches. However, it is unknown whether morphological computation actually differs between these different hierarchical levels of the control system. To investigate this, we simulated point-to-point and oscillatory human arm movements with a neuro-musculoskeletal model. We then quantify morphological computation on the different hierarchy levels. The results show that morphological computation is highest for the most central (highest) level of the modeled control hierarchy, where the movement initiation and timing are encoded. Furthermore, they show that the lowest neuronal control layer, the muscle stimulation input, exploits the morphological computation of the biochemical and biophysical muscle characteristics to generate smooth dynamic movements. This study provides evidence that the system's design in the mechanical as well as in the neurological structure can take over important contributions to control, which would otherwise need to be performed by the higher control levels.

A model for the transfer of control from the brain to the spinal cord through synaptic learning

The spinal cord is essential to the control of locomotion in legged animals and humans. However, the actual circuitry of the spinal controller remains only vaguely understood. Here we approach this problem from the viewpoint of learning. More precisely, we assume the circuitry evolves through the transfer of control from the brain to the spinal cord, propose a specific learning mechanism for this transfer based on the error between the cord and brain contributions to muscle control, and study the resulting structure of the spinal controller in a simplified neuromuscular model of human locomotion. The model focuses on the leg rebound behavior in stance and represents the spinal circuitry with 150 muscle reflexes. We find that after learning a spinal controller has evolved that produces leg rebound motions in the absence of a central brain input with only three structural reflex groups. These groups contain individual reflexes well known from physiological experiments but thought to serve separate purposes in the control of human locomotion. Our results suggest a more holistic interpretation of the role of individual sensory projections in spinal networks than is common. In addition, we discuss potential neural correlates for the proposed learning mechanism that may be probed in experiments. Together with such experiments, neuromuscular models of spinal learning likely will become effective tools for uncovering the structure and development of the spinal control circuitry.

Humans use multi-objective control to regulate lateral foot placement when walking

A fundamental question in human motor neuroscience is to determine how the nervous system generates goal-directed movements despite inherent physiological noise and redundancy. Walking exhibits considerable variability and equifinality of task solutions. Existing models of bipedal walking do not yet achieve both continuous dynamic balance control and the equifinality of foot placement humans exhibit. Appropriate computational models are critical to disambiguate the numerous possibilities of how to regulate stepping movements to achieve different walking goals. Here, we extend a theoretical and computational Goal Equivalent Manifold (GEM) framework to generate predictive models, each posing a different experimentally testable hypothesis. These models regulate stepping movements to achieve any of three hypothesized goals, either alone or in combination: maintain lateral position, maintain lateral speed or “heading”, and/or maintain step width. We compared model predictions against human experimental data. Uni-objective control models demonstrated clear redundancy between stepping variables, but could not replicate human stepping dynamics. Most multi-objective control models that balanced maintaining two of the three hypothesized goals also failed to replicate human stepping dynamics. However, multi-objective models that strongly prioritized regulating step width over lateral position did successfully replicate all of the relevant step-to-step dynamics observed in humans. Independent analyses confirmed this control was consistent with linear error correction and replicated step-to-step dynamics of individual foot placements. Thus, the regulation of lateral stepping movements is inherently multi-objective and balances task-specific trade-offs between competing task goals. To determine how people walk in their environment requires understanding both walking biomechanics and how the nervous system regulates movements from step-to-step. Analogous to mechanical “templates” of locomotor biomechanics, our models serve as “control templates” for how humans regulate stepping movements from each step to the next. These control templates are symbiotic with well-established mechanical templates, providing complimentary insights into walking regulation.

When we walk, we walk in real-world contexts and with specific goal to achieve. Side-to-side movements are paramount because walking bipeds (humans, animals, robots, etc.) are inherently more unstable laterally. This is particularly important in older adults as sideways falls greatly increase hip fracture risk. Additionally, we normally walk on paths that limit (more or less) our lateral movements. Appropriately regulating lateral stepping movements is thus critical to achieving successful locomotion in any such context. Here, we use appropriate models to test competing hypotheses about how humans regulate lateral stepping movements from each step to the next to identify what task goals they try to achieve. Our work both bridges and unifies perspectives from dynamic walking and computational motor control to provide a coherent theoretical and computational framework from which to study motor regulation in the context of goal-directedness across a wide range of walking tasks and/or conditions.

Autonomous Functional Movements in a Tendon-Driven Limb via Limited Experience

Robots will become ubiquitously useful only when they can use few attempts to teach themselves to perform different tasks, even with complex bodies and in dynamical environments. Vertebrates, in fact, use sparse trial-and-error to learn multiple tasks despite their intricate tendon-driven anatomies-which are particularly hard to control because they are simultaneously nonlinear, under-determined, and over-determined. We demonstrate-for the first time in simulation and hardware-how a model-free, open-loop approach allows few-shot autonomous learning to produce effective movements in a 3-tendon 2-joint limb. We use a short period of motor babbling (to create an initial inverse map) followed by building functional habits by reinforcing high-reward behavior and refinements of the inverse map in a movement's neighborhood. This biologically-plausible algorithm, which we call G2P (General-to-Particular), can potentially enable quick, robust and versatile adaptation in robots as well as shed light on the foundations of the enviable functional versatility of organisms.

Bipedal robotic walking control derived from analysis of human locomotion

This paper proposes the design of a bipedal robotic controller where the function between the sensory input and motor output is treated as a black box derived from human data. In order to achieve this, we investigated the causal relationship between ground contact information from the feet and leg muscle activity n human walking and calculated filter functions which transform sensory signals to motor actions. A minimal, nonlinear, and robust control system was created and subsequently analysed by applying it to our bipedal robot RunBot III without any central pattern generators or precise trajectory control. The results demonstrate that our controller can generate stable robotic walking. This indicates that complex locomotion patterns can result from a simple model based on reflexes and supports the premise that human-derived control strategies have potential applications in robotics or assistive devices.

Adaptive Control Strategies for Interlimb Coordination in Legged Robots: A Review

Walking animals produce adaptive interlimb coordination during locomotion in accordance with their situation. Interlimb coordination is generated through the dynamic interactions of the neural system, the musculoskeletal system, and the environment, although the underlying mechanisms remain unclear. Recently, investigations of the adaptation mechanisms of living beings have attracted attention, and bio-inspired control systems based on neurophysiological findings regarding sensorimotor interactions are being developed for legged robots. In this review, we introduce adaptive interlimb coordination for legged robots induced by various factors (locomotion speed, environmental situation, body properties, and task). In addition, we show characteristic properties of adaptive interlimb coordination, such as gait hysteresis and different time-scale adaptations. We also discuss the underlying mechanisms and control strategies to achieve adaptive interlimb coordination and the design principle for the control system of legged robots.

Self-Organized Behavior Generation for Musculoskeletal Robots

With the accelerated development of robot technologies, control becomes one of the central themes of research. In traditional approaches, the controller, by its internal functionality, finds appropriate actions on the basis of specific objectives for the task at hand. While very successful in many applications, self-organized control schemes seem to be favored in large complex systems with unknown dynamics or which are difficult to model. Reasons are the expected scalability, robustness, and resilience of self-organizing systems. The paper presents a self-learning neurocontroller based on extrinsic differential plasticity introduced recently, applying it to an anthropomorphic musculoskeletal robot arm with attached objects of unknown physical dynamics. The central finding of the paper is the following effect: by the mere feedback through the internal dynamics of the object, the robot is learning to relate each of the objects with a very specific sensorimotor pattern. Specifically, an attached pendulum pilots the arm into a circular motion, a half-filled bottle produces axis oriented shaking behavior, a wheel is getting rotated, and wiping patterns emerge automatically in a table-plus-brush setting. By these object-specific dynamical patterns, the robot may be said to recognize the object's identity, or in other words, it discovers dynamical affordances of objects. Furthermore, when including hand coordinates obtained from a camera, a dedicated hand-eye coordination self-organizes spontaneously. These phenomena are discussed from a specific dynamical system perspective. Central is the dedicated working regime at the border to instability with its potentially infinite reservoir of (limit cycle) attractors “waiting” to be excited. Besides converging toward one of these attractors, variate behavior is also arising from a self-induced attractor morphing driven by the learning rule. We claim that experimental investigations with this anthropomorphic, self-learning robot not only generate interesting and potentially useful behaviors, but may also help to better understand what subjective human muscle feelings are, how they can be rooted in sensorimotor patterns, and how these concepts may feed back on robotics.

A functional electrical stimulation system for human walking inspired by reflexive control principles

This study presents an innovative multichannel functional electrical stimulation gait-assist system which employs a well-established purely reflexive control algorithm, previously tested in a series of bipedal walking robots. In these robots, ground contact information was used to activate motors in the legs, generating a gait cycle similar to that of humans. Rather than developing a sophisticated closed-loop functional electrical stimulation control strategy for stepping, we have instead utilised our simple reflexive model where muscle activation is induced through transfer functions which translate sensory signals, predominantly ground contact information, into motor actions. The functionality of the functional electrical stimulation system was tested by analysis of the gait function of seven healthy volunteers during functional electrical stimulation-assisted treadmill walking compared to unassisted walking. The results demonstrated that the system was successful in synchronising muscle activation throughout the gait cycle and was able to promote functional hip and ankle movements. Overall, the study demonstrates the potential of human-inspired robotic systems in the design of assistive devices for bipedal walking.

An Adaptive Neural Mechanism for Acoustic Motion Perception with Varying Sparsity

Biological motion-sensitive neural circuits are quite adept in perceiving the relative motion of a relevant stimulus. Motion perception is a fundamental ability in neural sensory processing and crucial in target tracking tasks. Tracking a stimulus entails the ability to perceive its motion, i.e., extracting information about its direction and velocity. Here we focus on auditory motion perception of sound stimuli, which is poorly understood as compared to its visual counterpart. In earlier work we have developed a bio-inspired neural learning mechanism for acoustic motion perception. The mechanism extracts directional information via a model of the peripheral auditory system of lizards. The mechanism uses only this directional information obtained via specific motor behaviour to learn the angular velocity of unoccluded sound stimuli in motion. In nature however the stimulus being tracked may be occluded by artefacts in the environment, such as an escaping prey momentarily disappearing behind a cover of trees. This article extends the earlier work by presenting a comparative investigation of auditory motion perception for unoccluded and occluded tonal sound stimuli with a frequency of 2.2 kHz in both simulation and practice. Three instances of each stimulus are employed, differing in their movement velocities-0.5°/time step, 1.0°/time step and 1.5°/time step. To validate the approach in practice, we implement the proposed neural mechanism on a wheeled mobile robot and evaluate its performance in auditory tracking.

Adaptation mechanism of interlimb coordination in human split-belt treadmill walking through learning of foot contact timing: a robotics study

Human walking behaviour adaptation strategies have previously been examined using split-belt treadmills, which have two parallel independently controlled belts. In such human split-belt treadmill walking, two types of adaptations have been identified: early and late. Early-type adaptations appear as rapid changes in interlimb and intralimb coordination activities when the belt speeds of the treadmill change between tied (same speed for both belts) and split-belt (different speeds for each belt) configurations. By contrast, late-type adaptations occur after the early-type adaptations as a gradual change and only involve interlimb coordination. Furthermore, interlimb coordination shows after-effects that are related to these adaptations. It has been suggested that these adaptations are governed primarily by the spinal cord and cerebellum, but the underlying mechanism remains unclear. Because various physiological findings suggest that foot contact timing is crucial to adaptive locomotion, this paper reports on the development of a two-layered control model for walking composed of spinal and cerebellar models, and on its use as the focus of our control model. The spinal model generates rhythmic motor commands using an oscillator network based on a central pattern generator and modulates the commands formulated in immediate response to foot contact, while the cerebellar model modifies motor commands through learning based on error information related to differences between the predicted and actual foot contact timings of each leg. We investigated adaptive behaviour and its mechanism by split-belt treadmill walking experiments using both computer simulations and an experimental bipedal robot. Our results showed that the robot exhibited rapid changes in interlimb and intralimb coordination that were similar to the early-type adaptations observed in humans. In addition, despite the lack of direct interlimb coordination control, gradual changes and after-effects in the interlimb coordination appeared in a manner that was similar to the late-type adaptations and after-effects observed in humans. The adaptation results of the robot were then evaluated in comparison with human split-belt treadmill walking, and the adaptation mechanism was clarified from a dynamic viewpoint.

Integration of an adaptive swing control into a neuromuscular human walking model

Understanding the neuromuscular control underlying human locomotion has the potential to deliver practical controllers for humanoid and prosthetic robots. However, neurocontrollers developed in forward dynamic simulations are seldom applied as practical controllers due to their lack of robustness and adaptability. A key element for robust and adaptive locomotion is swing leg placement. Here we integrate a previously identified robust swing leg controller into a full neuromuscular human walking model and demonstrate that the integrated model has largely improved behaviors including walking on very rough terrain (±10 cm) and stair climbing (15 cm stairs). These initial results highlight the potential of the identified robust swing control. We plan to generalize it to a range of human locomotion behaviors critical in rehabilitation robotics.

Identifying stride-to-stride control strategies in human treadmill walking

Variability is ubiquitous in human movement, arising from internal and external noise, inherent biological redundancy, and from the neurophysiological control actions that help regulate movement fluctuations. Increased walking variability can lead to increased energetic cost and/or increased fall risk. Conversely, biological noise may be beneficial, even necessary, to enhance motor performance. Indeed, encouraging more variability actually facilitates greater improvements in some forms of locomotor rehabilitation. Thus, it is critical to identify the fundamental principles humans use to regulate stride-to-stride fluctuations in walking. This study sought to determine how humans regulate stride-to-stride fluctuations in stepping movements during treadmill walking. We developed computational models based on pre-defined goal functions to compare if subjects, from each stride to the next, tried to maintain the same speed as the treadmill, or instead stay in the same position on the treadmill. Both strategies predicted average behaviors empirically indistinguishable from each other and from that of humans. These strategies, however, predicted very different stride-to-stride fluctuation dynamics. Comparisons to experimental data showed that human stepping movements were generally well-predicted by the speed-control model, but not by the position-control model. Human subjects also exhibited no indications they corrected deviations in absolute position only intermittently: i.e., closer to the boundaries of the treadmill. Thus, humans clearly do not adopt a control strategy whose primary goal is to maintain some constant absolute position on the treadmill. Instead, humans appear to regulate their stepping movements in a way most consistent with a strategy whose primary goal is to try to maintain the same speed as the treadmill at each consecutive stride. These findings have important implications both for understanding how biological systems regulate walking in general and for being able to harness these mechanisms to develop more effective rehabilitation interventions to improve locomotor performance.

Neuromodulatory adaptive combination of correlation-based learning in cerebellum and reward-based learning in basal ganglia for goal-directed behavior control

Goal-directed decision making in biological systems is broadly based on associations between conditional and unconditional stimuli. This can be further classified as classical conditioning (correlation-based learning) and operant conditioning (reward-based learning). A number of computational and experimental studies have well established the role of the basal ganglia in reward-based learning, where as the cerebellum plays an important role in developing specific conditioned responses. Although viewed as distinct learning systems, recent animal experiments point toward their complementary role in behavioral learning, and also show the existence of substantial two-way communication between these two brain structures. Based on this notion of co-operative learning, in this paper we hypothesize that the basal ganglia and cerebellar learning systems work in parallel and interact with each other. We envision that such an interaction is influenced by reward modulated heterosynaptic plasticity (RMHP) rule at the thalamus, guiding the overall goal directed behavior. Using a recurrent neural network actor-critic model of the basal ganglia and a feed-forward correlation-based learning model of the cerebellum, we demonstrate that the RMHP rule can effectively balance the outcomes of the two learning systems. This is tested using simulated environments of increasing complexity with a four-wheeled robot in a foraging task in both static and dynamic configurations. Although modeled with a simplified level of biological abstraction, we clearly demonstrate that such a RMHP induced combinatorial learning mechanism, leads to stabler and faster learning of goal-directed behaviors, in comparison to the individual systems. Thus, in this paper we provide a computational model for adaptive combination of the basal ganglia and cerebellum learning systems by way of neuromodulated plasticity for goal-directed decision making in biological and bio-mimetic organisms.

Reflex control of robotic gait using human walking data

Control of human walking is not thoroughly understood, which has implications in developing suitable strategies for the retraining of a functional gait following neurological injuries such as spinal cord injury (SCI). Bipedal robots allow us to investigate simple elements of the complex nervous system to quantify their contribution to motor control. RunBot is a bipedal robot which operates through reflexes without using central pattern generators or trajectory planning algorithms. Ground contact information from the feet is used to activate motors in the legs, generating a gait cycle visually similar to that of humans. Rather than developing a more complicated biologically realistic neural system to control the robot's stepping, we have instead further simplified our model by measuring the correlation between heel contact and leg muscle activity (EMG) in human subjects during walking and from this data created filter functions transferring the sensory data into motor actions. Adaptive filtering was used to identify the unknown transfer functions which translate the contact information into muscle activation signals. Our results show a causal relationship between ground contact information from the heel and EMG, which allows us to create a minimal, linear, analogue control system for controlling walking. The derived transfer functions were applied to RunBot II as a proof of concept. The gait cycle produced was stable and controlled, which is a positive indication that the transfer functions have potential for use in the control of assistive devices for the retraining of an efficient and effective gait with potential applications in SCI rehabilitation.

Real-time Walking Pattern Generation for a Biped Robot with Hybrid CPG-ZMP Algorithm

Biped robots have better mobility than conventional wheeled robots. The bio-inspired method based on a central pattern generator (CPG) can be used to control biped robot walking in a manner like human beings. However, to achieve stable locomotion, it is difficult to modulate the parameters for the neural networks to coordinate every degree of freedom of the walking robot. The zero moment point (ZMP) method is very popular for the stability control of biped robot walking. However, the reference trajectories have low energy efficiency, lack naturalness and need significant offline calculation. This paper presents a new method for biped real-time walking generation using a hybrid CPG-ZMP control algorithm. The method can realize a stable walking pattern by combining the ZMP criterion with rhythmic motion control. The CPG component is designed to generate the desired motion for each robot joint, which is modulated by phase resetting according to foot contact information. By introducing the ZMP location, the activity of the CPG output signal is adjusted to coordinate the limbs’ motion and allow the robot to maintain balance during the process of locomotion. The numerical simulation results show that, compared with the CPG method, the new hybrid CPG-ZMP algorithm can enhance the robustness of the CPG parameters and improve the stability of the robot. In addition, the proposed algorithm is more energy efficient than the ZMP method. The results also demonstrate that the control system can generate an adaptive walking pattern through interactions between the robot, the CPG and the environment.

From spontaneous motor activity to coordinated behaviour: a developmental model

In mammals, the developmental path that links the primary behaviours observed during foetal stages to the full fledged behaviours observed in adults is still beyond our understanding. Often theories of motor control try to deal with the process of incremental learning in an abstract and modular way without establishing any correspondence with the mammalian developmental stages. In this paper, we propose a computational model that links three distinct behaviours which appear at three different stages of development. In order of appearance, these behaviours are: spontaneous motor activity (SMA), reflexes, and coordinated behaviours, such as locomotion. The goal of our model is to address in silico four hypotheses that are currently hard to verify in vivo: First, the hypothesis that spinal reflex circuits can be self-organized from the sensor and motor activity induced by SMA. Second, the hypothesis that supraspinal systems can modulate reflex circuits to achieve coordinated behaviour. Third, the hypothesis that, since SMA is observed in an organism throughout its entire lifetime, it provides a mechanism suitable to maintain the reflex circuits aligned with the musculoskeletal system, and thus adapt to changes in body morphology. And fourth, the hypothesis that by changing the modulation of the reflex circuits over time, one can switch between different coordinated behaviours. Our model is tested in a simulated musculoskeletal leg actuated by six muscles arranged in a number of different ways. Hopping is used as a case study of coordinated behaviour. Our results show that reflex circuits can be self-organized from SMA, and that, once these circuits are in place, they can be modulated to achieve coordinated behaviour. In addition, our results show that our model can naturally adapt to different morphological changes and perform behavioural transitions.

Multi-layered multi-pattern CPG for adaptive locomotion of humanoid robots

In this paper, we present an extended mathematical model of the central pattern generator (CPG) in the spinal cord. The proposed CPG model is used as the underlying low-level controller of a humanoid robot to generate various walking patterns. Such biological mechanisms have been demonstrated to be robust in locomotion of animal. Our model is supported by two neurophysiological studies. The first study identified a neural circuitry consisting of a two-layered CPG, in which pattern formation and rhythm generation are produced at different levels. The second study focused on a specific neural model that can generate different patterns, including oscillation. This neural model was employed in the pattern generation layer of our CPG, which enables it to produce different motion patterns-rhythmic as well as non-rhythmic motions. Due to the pattern-formation layer, the CPG is able to produce behaviors related to the dominating rhythm (extension/flexion) and rhythm deletion without rhythm resetting. The proposed multi-layered multi-pattern CPG model (MLMP-CPG) has been deployed in a 3D humanoid robot (NAO) while it performs locomotion tasks. The effectiveness of our model is demonstrated in simulations and through experimental results.

Behavior control in the sensorimotor loop with short-term synaptic dynamics induced by self-regulating neurons

The behavior and skills of living systems depend on the distributed control provided by specialized and highly recurrent neural networks. Learning and memory in these systems is mediated by a set of adaptation mechanisms, known collectively as neuronal plasticity. Translating principles of recurrent neural control and plasticity to artificial agents has seen major strides, but is usually hampered by the complex interactions between the agent's body and its environment. One of the important standing issues is for the agent to support multiple stable states of behavior, so that its behavioral repertoire matches the requirements imposed by these interactions. The agent also must have the capacity to switch between these states in time scales that are comparable to those by which sensory stimulation varies. Achieving this requires a mechanism of short-term memory that allows the neurocontroller to keep track of the recent history of its input, which finds its biological counterpart in short-term synaptic plasticity. This issue is approached here by deriving synaptic dynamics in recurrent neural networks. Neurons are introduced as self-regulating units with a rich repertoire of dynamics. They exhibit homeostatic properties for certain parameter domains, which result in a set of stable states and the required short-term memory. They can also operate as oscillators, which allow them to surpass the level of activity imposed by their homeostatic operation conditions. Neural systems endowed with the derived synaptic dynamics can be utilized for the neural behavior control of autonomous mobile agents. The resulting behavior depends also on the underlying network structure, which is either engineered or developed by evolutionary techniques. The effectiveness of these self-regulating units is demonstrated by controlling locomotion of a hexapod with 18 degrees of freedom, and obstacle-avoidance of a wheel-driven robot.

Visually guided gait modifications for stepping over an obstacle: a bio-inspired approach

There is an increasing interest in conceiving robotic systems that are able to move and act in an unstructured and not predefined environment, for which autonomy and adaptability are crucial features. In nature, animals are autonomous biological systems, which often serve as bio-inspiration models, not only for their physical and mechanical properties, but also their control structures that enable adaptability and autonomy-for which learning is (at least) partially responsible. This work proposes a system which seeks to enable a quadruped robot to online learn to detect and to avoid stumbling on an obstacle in its path. The detection relies in a forward internal model that estimates the robot's perceptive information by exploring the locomotion repetitive nature. The system adapts the locomotion in order to place the robot optimally before attempting to step over the obstacle, avoiding any stumbling. Locomotion adaptation is achieved by changing control parameters of a central pattern generator (CPG)-based locomotion controller. The mechanism learns the necessary alterations to the stride length in order to adapt the locomotion by changing the required CPG parameter. Both learning tasks occur online and together define a sensorimotor map, which enables the robot to learn to step over the obstacle in its path. Simulation results show the feasibility of the proposed approach.

Humanoids Learning to Walk: A Natural CPG-Actor-Critic Architecture

The identification of learning mechanisms for locomotion has been the subject of much research for some time but many challenges remain. Dynamic systems theory (DST) offers a novel approach to humanoid learning through environmental interaction. Reinforcement learning (RL) has offered a promising method to adaptively link the dynamic system to the environment it interacts with via a reward-based value system. In this paper, we propose a model that integrates the above perspectives and applies it to the case of a humanoid (NAO) robot learning to walk the ability of which emerges from its value-based interaction with the environment. In the model, a simplified central pattern generator (CPG) architecture inspired by neuroscientific research and DST is integrated with an actor-critic approach to RL (cpg-actor-critic). In the cpg-actor-critic architecture, least-square-temporal-difference based learning converges to the optimal solution quickly by using natural gradient learning and balancing exploration and exploitation. Futhermore, rather than using a traditional (designer-specified) reward it uses a dynamic value function as a stability indicator that adapts to the environment. The results obtained are analyzed using a novel DST-based embodied cognition approach. Learning to walk, from this perspective, is a process of integrating levels of sensorimotor activity and value.

Do humans optimally exploit redundancy to control step variability in walking?

It is widely accepted that humans and animals minimize energetic cost while walking. While such principles predict average behavior, they do not explain the variability observed in walking. For robust performance, walking movements must adapt at each step, not just on average. Here, we propose an analytical framework that reconciles issues of optimality, redundancy, and stochasticity. For human treadmill walking, we defined a goal function to formulate a precise mathematical definition of one possible control strategy: maintain constant speed at each stride. We recorded stride times and stride lengths from healthy subjects walking at five speeds. The specified goal function yielded a decomposition of stride-to-stride variations into new gait variables explicitly related to achieving the hypothesized strategy. Subjects exhibited greatly decreased variability for goal-relevant gait fluctuations directly related to achieving this strategy, but far greater variability for goal-irrelevant fluctuations. More importantly, humans immediately corrected goal-relevant deviations at each successive stride, while allowing goal-irrelevant deviations to persist across multiple strides. To demonstrate that this was not the only strategy people could have used to successfully accomplish the task, we created three surrogate data sets. Each tested a specific alternative hypothesis that subjects used a different strategy that made no reference to the hypothesized goal function. Humans did not adopt any of these viable alternative strategies. Finally, we developed a sequence of stochastic control models of stride-to-stride variability for walking, based on the Minimum Intervention Principle. We demonstrate that healthy humans are not precisely "optimal," but instead consistently slightly over-correct small deviations in walking speed at each stride. Our results reveal a new governing principle for regulating stride-to-stride fluctuations in human walking that acts independently of, but in parallel with, minimizing energetic cost. Thus, humans exploit task redundancies to achieve robust control while minimizing effort and allowing potentially beneficial motor variability.

Action and behavior: a free-energy formulation

We have previously tried to explain perceptual inference and learning under a free-energy principle that pursues Helmholtz's agenda to understand the brain in terms of energy minimization. It is fairly easy to show that making inferences about the causes of sensory data can be cast as the minimization of a free-energy bound on the likelihood of sensory inputs, given an internal model of how they were caused. In this article, we consider what would happen if the data themselves were sampled to minimize this bound. It transpires that the ensuing active sampling or inference is mandated by ergodic arguments based on the very existence of adaptive agents. Furthermore, it accounts for many aspects of motor behavior; from retinal stabilization to goal-seeking. In particular, it suggests that motor control can be understood as fulfilling prior expectations about proprioceptive sensations. This formulation can explain why adaptive behavior emerges in biological agents and suggests a simple alternative to optimal control theory. We illustrate these points using simulations of oculomotor control and then apply to same principles to cued and goal-directed movements. In short, the free-energy formulation may provide an alternative perspective on the motor control that places it in an intimate relationship with perception.

Prediction of plantar shear stress distribution by artificial intelligence methods

Shear forces under the human foot are thought to be responsible for various foot pathologies such as diabetic plantar ulcers and athletic blisters. Frictional shear forces might also play a role in the metatarsalgia observed among hallux valgus (HaV) and rheumatoid arthritis (RA) patients. Due to the absence of commercial devices capable of measuring shear stress distribution, a number of linear models were developed. All of these have met with limited success. This study used nonlinear methods, specifically neural network and fuzzy logic schemes, to predict the distribution of plantar shear forces based on vertical loading parameters. In total, 73 subjects were recruited; 17 had diabetic neuropathy, 14 had HaV, 9 had RA, 11 had frequent foot blisters, and 22 were healthy. A feed-forward neural network (NN) and adaptive neurofuzzy inference system (NFIS) were built. These systems were then applied to a custom-built platform, which collected plantar pressure and shear stress data as subjects walked over the device. The inputs to both models were peak pressure, peak pressure-time integral, and time to peak pressure, and the output was peak resultant shear. Root-mean-square error (RMSE) values were calculated to test the models' accuracy. RMSE/actual shear ratio varied between 0.27 and 0.40 for NN predictions. Similarly, NFIS estimations resulted in a 0.28-0.37 ratio for local peak values in all subject groups. On the other hand, error percentages for global peak shear values were found to be in the range 11.4-44.1. These results indicate that there is no direct relationship between pressure and shear magnitudes. Future research should aim to decrease error levels by introducing shear stress dependent variables into the models.

Generating spatiotemporal joint torque patterns from dynamical synchronization of distributed pattern generators

Pattern generators found in the spinal cord are no more seen as simple rhythmic oscillators for motion control. Indeed, they achieve flexible and dynamical coordination in interaction with the body and the environment dynamics giving to rise motor synergies. Discovering the mechanisms underlying the control of motor synergies constitutes an important research question not only for neuroscience but also for robotics: the motors coordination of high dimensional robotic systems is still a drawback and new control methods based on biological solutions may reduce their overall complexity. We propose to model the flexible combination of motor synergies in embodied systems via partial phase synchronization of distributed chaotic systems; for specific coupling strength, chaotic systems are able to phase synchronize their dynamics to the resonant frequencies of one external force. We take advantage of this property to explore and exploit the intrinsic dynamics of one specified embodied system. In two experiments with bipedal walkers, we show how motor synergies emerge when the controllers phase synchronize to the body's dynamics, entraining it to its intrinsic behavioral patterns. This stage is characterized by directed information flow from the sensors to the motors exhibiting the optimal situation when the body dynamics drive the controllers (mutual entrainment). Based on our results, we discuss the relevance of our findings for modeling the modular control of distributed pattern generators exhibited in the spinal cord, and for exploring the motor synergies in robots.

Reinforcement learning or active inference?

This paper questions the need for reinforcement learning or control theory when optimising behaviour. We show that it is fairly simple to teach an agent complicated and adaptive behaviours using a free-energy formulation of perception. In this formulation, agents adjust their internal states and sampling of the environment to minimize their free-energy. Such agents learn causal structure in the environment and sample it in an adaptive and self-supervised fashion. This results in behavioural policies that reproduce those optimised by reinforcement learning and dynamic programming. Critically, we do not need to invoke the notion of reward, value or utility. We illustrate these points by solving a benchmark problem in dynamic programming; namely the mountain-car problem, using active perception or inference under the free-energy principle. The ensuing proof-of-concept may be important because the free-energy formulation furnishes a unified account of both action and perception and may speak to a reappraisal of the role of dopamine in the brain.

Genetic algorithm-based optimal bipedal walking gait synthesis considering tradeoff between stability margin and speed

The inverse kinematics of a 12 degrees-of-freedom (DOFs) biped robot is formulated in terms of certain parameters. The biped walking gaits are developed using the parameters. The walking gaits are optimized using genetic algorithm (GA). The optimization is carried out considering relative importance of stability margin and walking speed. The stability margin depends on the position of zero-moment-point (ZMP) while walking speed varies with step-size. The ZMP is computed by an approximation-based method which does not require system dynamics. The optimal walking gaits are experimentally realized on a biped robot.

Mathematical properties of neuronal TD-rules and differential Hebbian learning: a comparison

A confusingly wide variety of temporally asymmetric learning rules exists related to reinforcement learning and/or to spike-timing dependent plasticity, many of which look exceedingly similar, while displaying strongly different behavior. These rules often find their use in control tasks, for example in robotics and for this rigorous convergence and numerical stability is required. The goal of this article is to review these rules and compare them to provide a better overview over their different properties. Two main classes will be discussed: temporal difference (TD) rules and correlation based (differential hebbian) rules and some transition cases. In general we will focus on neuronal implementations with changeable synaptic weights and a time-continuous representation of activity. In a machine learning (non-neuronal) context, for TD-learning a solid mathematical theory has existed since several years. This can partly be transferred to a neuronal framework, too. On the other hand, only now a more complete theory has also emerged for differential Hebb rules. In general rules differ by their convergence conditions and their numerical stability, which can lead to very undesirable behavior, when wanting to apply them. For TD, convergence can be enforced with a certain output condition assuring that the delta-error drops on average to zero (output control). Correlation based rules, on the other hand, converge when one input drops to zero (input control). Temporally asymmetric learning rules treat situations where incoming stimuli follow each other in time. Thus, it is necessary to remember the first stimulus to be able to relate it to the later occurring second one. To this end different types of so-called eligibility traces are being used by these two different types of rules. This aspect leads again to different properties of TD and differential Hebbian learning as discussed here. Thus, this paper, while also presenting several novel mathematical results, is mainly meant to provide a road map through the different neuronally emulated temporal asymmetrical learning rules and their behavior to provide some guidance for possible applications.