Continuous-Time Reinforcement Learning Control: A Review of Theoretical Results, Insights on Performance, and Needs for New Designs

This exposition discusses continuous-time reinforcement learning (CT-RL) for the control of affine nonlinear systems. We review four seminal methods that are the centerpieces of the most recent results on CT-RL control. We survey the theoretical results of the four methods, highlighting their fundamental importance and successes by including discussions on problem formulation, key assumptions, algorithm procedures, and theoretical guarantees. Subsequently, we evaluate the performance of the control designs to provide analyses and insights on the feasibility of these design methods for applications from a control designer's point of view. Through systematic evaluations, we point out when theory diverges from practical controller synthesis. We, furthermore, introduce a new quantitative analytical framework to diagnose the observed discrepancies. Based on the analyses and the insights gained through quantitative evaluations, we point out potential future research directions to unleash the potential of CT-RL control algorithms in addressing the identified challenges.

Adaptive optimal output regulation for wheel-legged robot Ollie: A data-driven approach

The dynamics of a robot may vary during operation due to both internal and external factors, such as non-ideal motor characteristics and unmodeled loads, which would lead to control performance deterioration and even instability. In this paper, the adaptive optimal output regulation (AOOR)-based controller is designed for the wheel-legged robot Ollie to deal with the possible model uncertainties and disturbances in a data-driven approach. We test the AOOR-based controller by forcing the robot to stand still, which is a conventional index to judge the balance controller for two-wheel robots. By online training with small data, the resultant AOOR achieves the optimality of the control performance and stabilizes the robot within a small displacement in rich experiments with different working conditions. Finally, the robot further balances a rolling cylindrical bottle on its top with the balance control using the AOOR, but it fails with the initial controller. Experimental results demonstrate that the AOOR-based controller shows the effectiveness and high robustness with model uncertainties and external disturbances.

Cooperative control strategy of wheel-legged robot based on attitude balance

To integrate the uneven terrain adaptivity of legged robots and the fast capacity of wheeled robots on even terrains, a four wheel-legged robot is addressed and the cooperative control strategy of wheels and legs based on attitude balance is investigated. Firstly, the kinematics of wheel-legged robot is analyzed, which contains the legged and wheeled motion modal. Secondly, the cooperative control strategy of wheel-legged robot based on attitude balance is proposed. The attitude is calculated by using the quaternion method and complementary filtering, and the attitude stability control of the wheel-legged robot is studied. The trajectory planning of leg motion including walk and trot gait is implemented, and the differential control of wheeled motion is deduced. And then, the cooperative motion control of wheels and legs is achieved by keeping the attitude balance of robot body. Finally, a small prototype is set up to validate the feasibility and effectiveness of proposed method. The experimental results show that the established wheel-legged robot can do walk, trot, and wheel-leg compound motion to overcome many complex terrains and environments.

A Novel Wheel-Legged Hexapod Robot

Traditional mobile robots are mainly divided into wheeled robots and legged robots. They have good performance at fast-moving speeds and crossing obstacles, and weak terrain adaptability and moving speeds, respectively. Combining the advantages of these two types mentioned, a multi-functional wheel-legged hexapod robot with strong climbing capacity was designed in this paper. Each wheel-leg of the robot is driven directly by a single motor and can move smoothly and quickly in a diagonal tripod gait. Based on the obstacle-crossing way of the wheel-leg and combined with the characteristics of insects moving stably in nature, the middle part of the robot body is wider than head and tail. Tripod gait was selected to control the robot locomotion. A series of simulations and experiments were conducted to validate its excellent adaptability to various environmental conditions. The robot can traverse rugged, broken, and obstacle-ridden ground and cross rugged surfaces full of obstacles without any terrain sensing or actively controlled adaptation. It can negotiate obstacles of approximately its own height, which is much higher than its centre of gravity range.

Upright and Crawling Locomotion and Its Transition for a Wheel-Legged Robot

To face the challenge of adapting to complex terrains and environments, we develop a novel wheel-legged robot that can switch motion modes to adapt to different environments. The robot can perform efficient and stable upright balanced locomotion on flat roads and flexible crawling in low and narrow passages. For passing through low and narrow passages, we propose a crawling motion control strategy and methods for transitioning between locomotion modes of wheel-legged robots. In practical applications, the smooth transition between the two motion modes is challenging. By optimizing the gravity work of the body, the optimal trajectory of the center of mass (CoM) for the transition from standing to crawling is obtained. By constructing and solving an optimization problem regarding the posture and motion trajectories of the underactuated model, the robot achieves a smooth transition from crawling to standing. In experiments, the wheel-legged robot successfully transitioned between the crawling mode and the upright balanced moving mode and flexibly passed a low and narrow passage. Consequently, the effectiveness of the control strategies and algorithms proposed in this paper are verified by experiments.