Decentralized control and local information for robust and adaptive decentralized Deep Reinforcement Learning

Neural dynamics of robust legged robots

Legged robot control has improved in recent years with the rise of deep reinforcement learning, however, much of the underlying neural mechanisms remain difficult to interpret. Our aim is to leverage bio-inspired methods from computational neuroscience to better understand the neural activity of robust robot locomotion controllers. Similar to past work, we observe that terrain-based curriculum learning improves agent stability. We study the biomechanical responses and neural activity within our neural network controller by simultaneously pairing physical disturbances with targeted neural ablations. We identify an agile hip reflex that enables the robot to regain its balance and recover from lateral perturbations. Model gradients are employed to quantify the relative degree that various sensory feedback channels drive this reflexive behavior. We also find recurrent dynamics are implicated in robust behavior, and utilize sampling-based ablation methods to identify these key neurons. Our framework combines model-based and sampling-based methods for drawing causal relationships between neural network activity and robust embodied robot behavior.

Predictive hierarchical reinforcement learning for path-efficient mapless navigation with moving target

Deep reinforcement learning (DRL) has been proven as a powerful approach for robot navigation over the past few years. DRL-based navigation does not require the pre-construction of a map, instead, high-performance navigation skills can be learned from trial-and-error experiences. However, recent DRL-based approaches mostly focus on a fixed navigation target. It is noted that when navigating to a moving target without maps, the performance of the standard RL structure drops dramatically on both the success rate and path efficiency. To address the mapless navigation problem with moving target, the predictive hierarchical DRL (pH-DRL) framework is proposed by integrating the long-term trajectory prediction to provide a cost-effective solution. In the proposed framework, the lower-level policy of the RL agent learns robot control actions to a specified goal, and the higher-level policy learns to make long-range planning of shorter navigation routes by sufficiently exploiting the predicted trajectories. By means of making decisions over two level of policies, the pH-DRL framework is robust to the unavoidable errors in long-term predictions. With the application of deep deterministic policy gradient (DDPG) for policy optimization, the pH-DDPG algorithm is developed based on the pH-DRL structure. Finally, through comparative experiments on the Gazebo simulator with several variants of the DDPG algorithm, the results demonstrate that the pH-DDPG outperforms other algorithms and achieves a high success rate and efficiency even though the target moves fast and randomly.

Having multiple selves helps learning agents explore and adapt in complex changing worlds

Satisfying a variety of conflicting needs in a changing environment is a fundamental challenge for any adaptive agent. Here, we show that designing an agent in a modular fashion as a collection of subagents, each dedicated to a separate need, powerfully enhanced the agent's capacity to satisfy its overall needs. We used the formalism of deep reinforcement learning to investigate a biologically relevant multiobjective task: continually maintaining homeostasis of a set of physiologic variables. We then conducted simulations in a variety of environments and compared how modular agents performed relative to standard monolithic agents (i.e., agents that aimed to satisfy all needs in an integrated manner using a single aggregate measure of success). Simulations revealed that modular agents a) exhibited a form of exploration that was intrinsic and emergent rather than extrinsically imposed; b) were robust to changes in nonstationary environments, and c) scaled gracefully in their ability to maintain homeostasis as the number of conflicting objectives increased. Supporting analysis suggested that the robustness to changing environments and increasing numbers of needs were due to intrinsic exploration and efficiency of representation afforded by the modular architecture. These results suggest that the normative principles by which agents have adapted to complex changing environments may also explain why humans have long been described as consisting of "multiple selves."

neuroWalknet, a controller for hexapod walking allowing for context dependent behavior

Decentralized control has been established as a key control principle in insect walking and has been successfully leveraged to account for a wide range of walking behaviors in the proposed neuroWalknet architecture. This controller allows for walking patterns at different velocities in both, forward and backward direction-quite similar to the behavior shown in stick insects-, for negotiation of curves, and for robustly dealing with various disturbances. While these simulations focus on the cooperation of different, decentrally controlled legs, here we consider a set of biological experiments not yet been tested by neuroWalknet, that focus on the function of the individual leg and are context dependent. These intraleg studies deal with four groups of interjoint reflexes. The reflexes are elicited by stimulation of the femoral chordotonal organ (fCO) or groups of campaniform sensilla (CS). Motor output signals are recorded from the alpha-joint, the beta-joint or the gamma-joint of the leg. Furthermore, the influence of these sensory inputs to artificially induced oscillations by application of pilocarpine has been studied. Although these biological data represent results obtained from different local reflexes in different contexts, they fit with and are embedded into the behavior shown by the global structure of neuroWalknet. In particular, a specific and intensively studied behavior, active reaction, has since long been assumed to represent a separate behavioral element, from which it is not clear why it occurs in some situations, but not in others. This question could now be explained as an emergent property of the holistic structure of neuroWalknet which has shown to be able to produce artificially elicited pilocarpine-driven oscillation that can be controlled by sensory input without the need of explicit innate CPG structures. As the simulation data result from a holistic system, further results were obtained that could be used as predictions to be tested in further biological experiments.

Modularity in Nervous Systems—a Key to Efficient Adaptivity for Deep Reinforcement Learning

Modularity as observed in biological systems has proven valuable for guiding classical motor theories towards good answers about action selection and execution. New challenges arise when we turn to learning: Trying to scale current computational models, such as deep reinforcement learning (DRL), to action spaces, input dimensions, and time horizons seen in biological systems still faces severe obstacles unless vast amounts of training data are available. This leads to the question: does biological modularity also hold an important key for better answers to obtain efficient adaptivity for deep reinforcement learning? We review biological experimental work on modularity in biological motor control and link this with current examples of (deep) RL approaches. Analyzing outcomes of simulation studies, we show that these approaches benefit from forms of modularization as found in biological systems. We identify three different strands of modularity exhibited in biological control systems. Two of them—modularity in state (i) and in action (ii) spaces—appear as a consequence of local interconnectivity (as in reflexes) and are often modulated by higher levels in a control hierarchy. A third strand arises from chunking of action elements along a (iii) temporal dimension. Usually interacting in an overarching spatio-temporal hierarchy of the overall system, the three strands offer major “factors” decomposing the entire modularity structure. We conclude that modularity with its above strands can provide an effective prior for DRL approaches to speed up learning considerably and making learned controllers more robust and adaptive.