Multi-step heuristic dynamic programming for optimal control of nonlinear discrete-time systems

Evolving and Incremental Value Iteration Schemes for Nonlinear Discrete-Time Zero-Sum Games

In this article, evolving and incremental value iteration (VI) frameworks are constructed to address the discrete-time zero-sum game problem. First, the evolving scheme means that the closed-loop system is regulated by using the evolving policy pair. During the control stage, we are committed to establishing the stability criterion in order to guarantee the availability of evolving policy pairs. Second, a novel incremental VI algorithm, which takes the historical information of the iterative process into account, is developed to solve the regulation and tracking problems for the nonlinear zero-sum game. Via introducing different incremental factors, it is highlighted that we can adjust the convergence rate of the iterative cost function sequence. Finally, two simulation examples, including linear and nonlinear systems, are conducted to demonstrate the performance and the validity of the proposed evolving and incremental VI schemes.

Event-Triggered Control of Nonlinear Discrete-Time System With Unknown Dynamics Based on HDP(λ)

The heuristic dynamic programming (HDP) ( λ )-based optimal control strategy, which takes a long-term prediction parameter λ into account using an iterative manner, accelerates the learning rate obviously. The computation complexity caused by the state-associated extra variable in λ -return value computing of the traditional value-gradient learning method can be reduced. However, as the iteration number increases, calculation costs have grown dramatically that bring huge challenge for the optimal control process with limited bandwidth and computational units. In this article, we propose an event-triggered HDP (ETHDP) ( λ ) optimal control strategy for nonlinear discrete-time (NDT) systems with unknown dynamics. The iterative relation for λ -return of the final target value is derived first. The event-triggered condition ensuring system stability is designed to reduce the computation and communication requirements. Next, we build a model-actor-critic neural network (NN) structure, in which the model NN evaluates the system state for getting λ -return of the current time target value, which is used to obtain the critic NN real-time update errors. The event-triggered optimal control signal and one-step-return value are approximated by actor and critic NN, respectively. Then, the event trigger-based uniformly ultimately bounded (UUB) stability of the system state and NN weight errors are demonstrated by applying the Lyapunov technology. Finally, we illustrate the effectiveness of our proposed ETHDP ( λ ) strategy by two cases.

An Improved Calibration Method for the IMU Biases Utilizing KF-Based AdaGrad Algorithm

In the field of high accuracy strapdown inertial navigation system (SINS), the inertial measurement unit (IMU) biases can severely affect the navigation accuracy. Traditionally we use Kalman filter (KF) to estimate those biases. However, KF is an unbiased estimation method based on the assumption of Gaussian white noise (GWN) while IMU sensors noise is irregular. Kalman filtering will no longer be accurate when the sensor’s noise is irregular. In order to obtain the optimal solution of the IMU biases, this paper proposes a novel method for the calibration of IMU biases utilizing the KF-based AdaGrad algorithm to solve this problem. Three improvements were made as the following: (1) The adaptive subgradient method (AdaGrad) is proposed to overcome the difficulty of setting step size. (2) A KF-based AdaGrad numerical function is derived and (3) a KF-based AdaGrad calibration algorithm is proposed in this paper. Experimental results show that the method proposed in this paper can effectively improve the accuracy of IMU biases in both static tests and car-mounted field tests.

Towards a Smarter Energy Management System for Hybrid Vehicles: A Comprehensive Review of Control Strategies

This paper presents a comprehensive review of energy management control strategies utilized in hybrid electric vehicles (HEVs). These can be categorized as rule-based strategies and optimization-based strategies. Rule-based strategies, as the most basic strategy, are widely used due to their simplicity and practical application. The focus of rule-based strategies is to determine and optimize the optimal threshold for mode switching; however, they fall into a local optimal solutions. To have better performance in energy management, optimization-based strategies were developed. The categories of the existing optimization-based strategies are identified from the latest literature, and a brief study of each strategy is discussed, which consists of the main research ideas, the research focus, advantages, disadvantages and improvements to ameliorate optimality and real-time performance. Deterministic dynamic programming strategy is regarded as a benchmark. Based on neural network and the large data processing technology, data-driven strategies are put forward due to their approximate optimality and high computational efficiency. Finally, the comprehensive performance of each control strategy is analyzed with respect to five aspects. This paper not only provides a comprehensive analysis of energy management control strategies for HEVs, but also presents the emphasis in the future.

Adaptive Constrained Optimal Control Design for Data-Based Nonlinear Discrete-Time Systems With Critic-Only Structure

Reinforcement learning has proved to be a powerful tool to solve optimal control problems over the past few years. However, the data-based constrained optimal control problem of nonaffine nonlinear discrete-time systems has rarely been studied yet. To solve this problem, an adaptive optimal control approach is developed by using the value iteration-based Q-learning (VIQL) with the critic-only structure. Most of the existing constrained control methods require the use of a certain performance index and only suit for linear or affine nonlinear systems, which is unreasonable in practice. To overcome this problem, the system transformation is first introduced with the general performance index. Then, the constrained optimal control problem is converted to an unconstrained optimal control problem. By introducing the action-state value function, i.e., Q-function, the VIQL algorithm is proposed to learn the optimal Q-function of the data-based unconstrained optimal control problem. The convergence results of the VIQL algorithm are established with an easy-to-realize initial condition . To implement the VIQL algorithm, the critic-only structure is developed, where only one neural network is required to approximate the Q-function. The converged Q-function obtained from the critic-only VIQL method is employed to design the adaptive constrained optimal controller based on the gradient descent scheme. Finally, the effectiveness of the developed adaptive control method is tested on three examples with computer simulation.

Real-time discrete suboptimal control for systems with input and state delays: Experimental tests on a dehydration process

This article presents a suboptimal control strategy with finite horizon for affine nonlinear discrete systems with both state and input delays. The Dynamic Programming Approach is used to obtain the suboptimal control sequence, but in order to avoid the computation of the Bellman functional, a numerical approximation of this function is proposed in every step. The feasibility of our proposal is demonstrated via an experimental test on a dehydration process and the obtained results show a good performance and behavior of this process. Then in order to demonstrate the benefits of using this kind of control strategy, the results are compared with a non optimal control strategy, particularly with respect to results produced by an industrial Proportional Integral Derivative (PID) Honeywell controller, which is tuned using the Ziegler-Nichols method.