1
|
Wallace BA, Si J. Continuous-Time Reinforcement Learning Control: A Review of Theoretical Results, Insights on Performance, and Needs for New Designs. IEEE TRANSACTIONS ON NEURAL NETWORKS AND LEARNING SYSTEMS 2024; 35:10199-10219. [PMID: 37027747 DOI: 10.1109/tnnls.2023.3245980] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Subscribe] [Scholar Register] [Indexed: 06/19/2023]
Abstract
This exposition discusses continuous-time reinforcement learning (CT-RL) for the control of affine nonlinear systems. We review four seminal methods that are the centerpieces of the most recent results on CT-RL control. We survey the theoretical results of the four methods, highlighting their fundamental importance and successes by including discussions on problem formulation, key assumptions, algorithm procedures, and theoretical guarantees. Subsequently, we evaluate the performance of the control designs to provide analyses and insights on the feasibility of these design methods for applications from a control designer's point of view. Through systematic evaluations, we point out when theory diverges from practical controller synthesis. We, furthermore, introduce a new quantitative analytical framework to diagnose the observed discrepancies. Based on the analyses and the insights gained through quantitative evaluations, we point out potential future research directions to unleash the potential of CT-RL control algorithms in addressing the identified challenges.
Collapse
|
2
|
Zhang J, Li Z, Wang S, Dai Y, Zhang R, Lai J, Zhang D, Chen K, Hu J, Gao W, Tang J, Zheng Y. Adaptive optimal output regulation for wheel-legged robot Ollie: A data-driven approach. Front Neurorobot 2023; 16:1102259. [PMID: 36733906 PMCID: PMC9888428 DOI: 10.3389/fnbot.2022.1102259] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Key Words] [Track Full Text] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 11/18/2022] [Accepted: 12/14/2022] [Indexed: 01/13/2023] Open
Abstract
The dynamics of a robot may vary during operation due to both internal and external factors, such as non-ideal motor characteristics and unmodeled loads, which would lead to control performance deterioration and even instability. In this paper, the adaptive optimal output regulation (AOOR)-based controller is designed for the wheel-legged robot Ollie to deal with the possible model uncertainties and disturbances in a data-driven approach. We test the AOOR-based controller by forcing the robot to stand still, which is a conventional index to judge the balance controller for two-wheel robots. By online training with small data, the resultant AOOR achieves the optimality of the control performance and stabilizes the robot within a small displacement in rich experiments with different working conditions. Finally, the robot further balances a rolling cylindrical bottle on its top with the balance control using the AOOR, but it fails with the initial controller. Experimental results demonstrate that the AOOR-based controller shows the effectiveness and high robustness with model uncertainties and external disturbances.
Collapse
Affiliation(s)
- Jingfan Zhang
- Tencent Robotics X, Tencent Holdings, Shenzhen, Guangdong, China
| | - Zhaoxiang Li
- School of Computer Science, Yangtze University, Jingzhou, Hubei, China
| | - Shuai Wang
- Tencent Robotics X, Tencent Holdings, Shenzhen, Guangdong, China,*Correspondence: Shuai Wang ✉
| | - Yuan Dai
- Tencent Robotics X, Tencent Holdings, Shenzhen, Guangdong, China
| | - Ruirui Zhang
- Tencent Robotics X, Tencent Holdings, Shenzhen, Guangdong, China
| | - Jie Lai
- Tencent Robotics X, Tencent Holdings, Shenzhen, Guangdong, China
| | - Dongsheng Zhang
- Tencent Robotics X, Tencent Holdings, Shenzhen, Guangdong, China
| | - Ke Chen
- Tencent Robotics X, Tencent Holdings, Shenzhen, Guangdong, China
| | - Jie Hu
- School of Computer Science, Yangtze University, Jingzhou, Hubei, China
| | - Weinan Gao
- State Key Laboratory of Synthetical Automation for Process Industries, Northeastern University, Shenyang, China
| | - Jianshi Tang
- School of Integrated Circuits, Tsinghua University, Beijing, China
| | - Yu Zheng
- Tencent Robotics X, Tencent Holdings, Shenzhen, Guangdong, China
| |
Collapse
|
3
|
Optimal 3-dimension trajectory-tracking guidance for reusable launch vehicle based on back-stepping adaptive dynamic programming. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07972-1] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
|
4
|
Cooperative control strategy of wheel-legged robot based on attitude balance. ROBOTICA 2022. [DOI: 10.1017/s0263574722001436] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/06/2022]
Abstract
Abstract
To integrate the uneven terrain adaptivity of legged robots and the fast capacity of wheeled robots on even terrains, a four wheel-legged robot is addressed and the cooperative control strategy of wheels and legs based on attitude balance is investigated. Firstly, the kinematics of wheel-legged robot is analyzed, which contains the legged and wheeled motion modal. Secondly, the cooperative control strategy of wheel-legged robot based on attitude balance is proposed. The attitude is calculated by using the quaternion method and complementary filtering, and the attitude stability control of the wheel-legged robot is studied. The trajectory planning of leg motion including walk and trot gait is implemented, and the differential control of wheeled motion is deduced. And then, the cooperative motion control of wheels and legs is achieved by keeping the attitude balance of robot body. Finally, a small prototype is set up to validate the feasibility and effectiveness of proposed method. The experimental results show that the established wheel-legged robot can do walk, trot, and wheel-leg compound motion to overcome many complex terrains and environments.
Collapse
|
5
|
A Novel Wheel-Legged Hexapod Robot. Biomimetics (Basel) 2022; 7:biomimetics7040146. [PMID: 36278703 PMCID: PMC9589936 DOI: 10.3390/biomimetics7040146] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Journal Information] [Subscribe] [Scholar Register] [Received: 08/01/2022] [Revised: 09/25/2022] [Accepted: 09/26/2022] [Indexed: 11/16/2022] Open
Abstract
Traditional mobile robots are mainly divided into wheeled robots and legged robots. They have good performance at fast-moving speeds and crossing obstacles, and weak terrain adaptability and moving speeds, respectively. Combining the advantages of these two types mentioned, a multi-functional wheel-legged hexapod robot with strong climbing capacity was designed in this paper. Each wheel-leg of the robot is driven directly by a single motor and can move smoothly and quickly in a diagonal tripod gait. Based on the obstacle-crossing way of the wheel-leg and combined with the characteristics of insects moving stably in nature, the middle part of the robot body is wider than head and tail. Tripod gait was selected to control the robot locomotion. A series of simulations and experiments were conducted to validate its excellent adaptability to various environmental conditions. The robot can traverse rugged, broken, and obstacle-ridden ground and cross rugged surfaces full of obstacles without any terrain sensing or actively controlled adaptation. It can negotiate obstacles of approximately its own height, which is much higher than its centre of gravity range.
Collapse
|
6
|
Qiu X, Yu Z, Meng L, Chen X, Zhao L, Huang G, Meng F. Upright and Crawling Locomotion and Its Transition for a Wheel-Legged Robot. MICROMACHINES 2022; 13:mi13081252. [PMID: 36014174 PMCID: PMC9414717 DOI: 10.3390/mi13081252] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Abstract] [Track Full Text] [Download PDF] [Figures] [Subscribe] [Scholar Register] [Received: 06/18/2022] [Revised: 07/26/2022] [Accepted: 07/30/2022] [Indexed: 12/04/2022]
Abstract
To face the challenge of adapting to complex terrains and environments, we develop a novel wheel-legged robot that can switch motion modes to adapt to different environments. The robot can perform efficient and stable upright balanced locomotion on flat roads and flexible crawling in low and narrow passages. For passing through low and narrow passages, we propose a crawling motion control strategy and methods for transitioning between locomotion modes of wheel-legged robots. In practical applications, the smooth transition between the two motion modes is challenging. By optimizing the gravity work of the body, the optimal trajectory of the center of mass (CoM) for the transition from standing to crawling is obtained. By constructing and solving an optimization problem regarding the posture and motion trajectories of the underactuated model, the robot achieves a smooth transition from crawling to standing. In experiments, the wheel-legged robot successfully transitioned between the crawling mode and the upright balanced moving mode and flexibly passed a low and narrow passage. Consequently, the effectiveness of the control strategies and algorithms proposed in this paper are verified by experiments.
Collapse
Affiliation(s)
- Xuejian Qiu
- School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China
| | - Zhangguo Yu
- School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China
- Key Laboratory of Biomimetic Robots and Systems, Ministry of Education, Beijing 100081, China
| | - Libo Meng
- School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China
- Key Laboratory of Biomimetic Robots and Systems, Ministry of Education, Beijing 100081, China
- Correspondence:
| | - Xuechao Chen
- School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China
- Key Laboratory of Biomimetic Robots and Systems, Ministry of Education, Beijing 100081, China
| | - Lingxuan Zhao
- School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China
| | - Gao Huang
- Faculty of Information Technology, Beijing University of Technology, Beijing 100124, China
| | - Fei Meng
- School of Mechatronical Engineering, Beijing Institute of Technology, Beijing 100081, China
- Key Laboratory of Biomimetic Robots and Systems, Ministry of Education, Beijing 100081, China
| |
Collapse
|
7
|
Wang X, Quan Z, Li Y, Liu Y. Event-triggered trajectory-tracking guidance for reusable launch vehicle based on neural adaptive dynamic programming. Neural Comput Appl 2022. [DOI: 10.1007/s00521-022-07468-y] [Citation(s) in RCA: 0] [Impact Index Per Article: 0] [Reference Citation Analysis] [Track Full Text] [Journal Information] [Subscribe] [Scholar Register] [Indexed: 11/29/2022]
|