How to train a self-driving vehicle: On the added value (or lack thereof) of curriculum learning and replay buffers

Learning from only real-world collected data can be unrealistic and time consuming in many scenario. One alternative is to use synthetic data as learning environments to learn rare situations and replay buffers to speed up the learning. In this work, we examine the hypothesis of how the creation of the environment affects the training of reinforcement learning agent through auto-generated environment mechanisms. We take the autonomous vehicle as an application. We compare the effect of two approaches to generate training data for artificial cognitive agents. We consider the added value of curriculum learning-just as in human learning-as a way to structure novel training data that the agent has not seen before as well as that of using a replay buffer to train further on data the agent has seen before. In other words, the focus of this paper is on characteristics of the training data rather than on learning algorithms. We therefore use two tasks that are commonly trained early on in autonomous vehicle research: lane keeping and pedestrian avoidance. Our main results show that curriculum learning indeed offers an additional benefit over a vanilla reinforcement learning approach (using Deep-Q Learning), but the replay buffer actually has a detrimental effect in most (but not all) combinations of data generation approaches we considered here. The benefit of curriculum learning does depend on the existence of a well-defined difficulty metric with which various training scenarios can be ordered. In the lane-keeping task, we can define it as a function of the curvature of the road, in which the steeper and more occurring curves on the road, the more difficult it gets. Defining such a difficulty metric in other scenarios is not always trivial. In general, the results of this paper emphasize both the importance of considering data characterization, such as curriculum learning, and the importance of defining an appropriate metric for the task.

Toward navigation ability for autonomous mobile robots with learning from demonstration paradigm

Learning from demonstration, as an important component of imitation learning, is a paradigm for robot to learn new tasks. Considering the application of learning from demonstration in the navigation issue, the robot can also acquire the navigation task via the human teacher’s demonstration. Based on research of the human brain neocortex, in this article, we present a learning from demonstration navigation paradigm from the perspective of hierarchical temporal memory theory. As a type of end-to-end learning form, the demonstrated relationship between perception data and motion commands will be learned and predicted by using hierarchical temporal memory. This framework first perceives images to obtain the corresponding categories information; then the categories incorporated with depth and motion command data are encoded as a sequence of sparse distributed representation vectors. The sequential vectors are treated as the inputs to train the navigation hierarchical temporal memory. After the training, the navigation hierarchical temporal memory stores the transitions of the perceived images, depth, and motion data so that future motion commands can be predicted. The performance of the proposed navigation strategy is evaluated via the real experiments and the public data sets.

Biofunctional Understanding and Conceptual Control: Searching for Systematic Consensus in Systemic Cohesion

For first generation scientists after the cognitive revolution, knowers were in active control over all (stages of) information processing. Then, following a decade of transition shaped by intense controversy, embodied cognition emerged and suggested sources of control other than those implied by metaphysical information processing. With a thematic focus on embodiment science and an eye toward systematic consensus in systemic cohesion, the present study explores the roles of biofunctional and conceptual control processes in the wholetheme spiral of biofunctional understanding (see Iran-Nejad and Irannejad, 2017b, Figure 1). According to this spiral, each of the two kinds of understanding has its own unique set of knower control processes. For conceptual understanding (CU), knowers have deliberate attention-allocation control over their first-person "knowthat" and "knowhow" content combined as mutually coherent corequisites. For biofunctional understanding (BU), knowers have attention-allocation control only over their knowthat content but knowhow control content is ordinarily conspicuously absent. To test the hypothesis of differences in the manner of control between CU and BU, participants in two experiments read identical-format statements for internal consistency, as response time was recorded. The results of Experiment 1 supported the hypothesis of differences in the manner of control between the two types of control processes; and Experiment 2 confirmed the results of Experiment 1. These findings are discussed in terms of the predicted differences between BU and CU control processes, their roles in regulating the physically unobservable flow of systemic cohesion in the wholetheme spiral, and a proposal for systematic consensus in systemic cohesion to serve as the second guiding principle in biofunctional embodiment science next to physical science's first guiding principle of systematic observation.