Virtual State Feedback Reference Tuning and Value Iteration Reinforcement Learning for Unknown Observable Systems Control

Trajectory Tracking within a Hierarchical Primitive-Based Learning Approach

A hierarchical learning control framework (HLF) has been validated on two affordable control laboratories: an active temperature control system (ATCS) and an electrical rheostatic braking system (EBS). The proposed HLF is data-driven and model-free, while being applicable on general control tracking tasks which are omnipresent. At the lowermost level, L1, virtual state-feedback control is learned from input–output data, using a recently proposed virtual state-feedback reference tuning (VSFRT) principle. L1 ensures a linear reference model tracking (or matching) and thus, indirect closed-loop control system (CLCS) linearization. On top of L1, an experiment-driven model-free iterative learning control (EDMFILC) is then applied for learning reference input–controlled outputs pairs, coined as primitives. The primitives’ signals at the L2 level encode the CLCS dynamics, which are not explicitly used in the learning phase. Data reusability is applied to derive monotonic and safely guaranteed learning convergence. The learning primitives in the L2 level are finally used in the uppermost and final L3 level, where a decomposition/recomposition operation enables prediction of the optimal reference input assuring optimal tracking of a previously unseen trajectory, without relearning by repetitions, as it was in level L2. Hence, the HLF enables control systems to generalize their tracking behavior to new scenarios by extrapolating their current knowledge base. The proposed HLF framework endows the CLCSs with learning, memorization and generalization features which are specific to intelligent organisms. This may be considered as an advancement towards intelligent, generalizable and adaptive control systems.

Model Reference Tracking Control Solutions for a Visual Servo System Based on a Virtual State from Unknown Dynamics

This paper focuses on validating a model-free Value Iteration Reinforcement Learning (MFVI-RL) control solution on a visual servo tracking system in a comprehensive manner starting from theoretical convergence analysis to detailed hardware and software implementation. Learning is based on a virtual state representation reconstructed from input-output (I/O) system samples under nonlinear observability and unknown dynamics assumptions, while the goal is to ensure linear output reference model (ORM) tracking. Secondary, a competitive model-free Virtual State-Feedback Reference Tuning (VSFRT) is learned from the same I/O data using the same virtual state representation, demonstrating the framework’s learning capability. A model-based two degrees-of-freedom (2DOF) output feedback controller serving as a comparisons baseline is designed and tuned using an identified system model. With similar complexity and linear controller structure, MFVI-RL is shown to be superior, confirming that the model-based design issue of poor identified system model and control performance degradation can be solved in a direct data-driven style. Apart from establishing a formal connection between output feedback control, state feedback control and also between classical control and artificial intelligence methods, the results also point out several practical trade-offs, such as I/O data exploration quality and control performance leverage with data volume, control goal and controller complexity.

Hierarchical Cognitive Control for Unknown Dynamic Systems Tracking

A general control system tracking learning framework is proposed, by which an optimal learned tracking behavior called ‘primitive’ is extrapolated to new unseen trajectories without requiring relearning. This is considered intelligent behavior and strongly related to the neuro-motor cognitive control of biological (human-like) systems that deliver suboptimal executions for tasks outside of their current knowledge base, by using previously memorized experience. However, biological systems do not solve explicit mathematical equations for solving learning and prediction tasks. This stimulates the proposed hierarchical cognitive-like learning framework, based on state-of-the-art model-free control: (1) at the low-level L1, an approximated iterative Value Iteration for linearizing the closed-loop system (CLS) behavior by a linear reference model output tracking is first employed; (2) an experiment-driven Iterative Learning Control (EDILC) applied to the CLS from the reference input to the controlled output learns simple tracking tasks called ‘primitives’ in the secondary L2 level, and (3) the tertiary level L3 extrapolates the primitives’ optimal tracking behavior to new tracking tasks without trial-based relearning. The learning framework relies only on input-output system data to build a virtual state space representation of the underlying controlled system that is assumed to be observable. It has been shown to be effective by experimental validation on a representative, coupled, nonlinear, multivariable real-world system. Able to cope with new unseen scenarios in an optimal fashion, the hierarchical learning framework is an advance toward cognitive control systems.

A Multi-Dimensional Goal Aircraft Guidance Approach Based on Reinforcement Learning with a Reward Shaping Algorithm

Guiding an aircraft to 4D waypoints at a certain heading is a multi-dimensional goal aircraft guidance problem. In order to improve the performance and solve this problem, this paper proposes a multi-layer RL approach. The approach enables the autopilot in an ATC simulator to guide an aircraft to 4D waypoints at certain latitude, longitude, altitude, heading, and arrival time, respectively. To be specific, a multi-layer RL approach is proposed to simplify the neural network structure and reduce the state dimensions. A shaped reward function that involves the potential function and Dubins path method is applied. Experimental and simulation results show that the proposed approach can significantly improve the convergence efficiency and trajectory performance. Furthermore, the results indicate possible application prospects in team aircraft guidance tasks, since the aircraft can directly approach a goal without waiting in a specific pattern, thereby overcoming the problem of current ATC simulators.

Robust Model Reference Adaptive Control for Tail-Sitter VTOL Aircraft

This study presents a control design of roll motion for a vertical take-off and landing unmanned air vehicle (VTOL-UAV) design based on the Model Reference Adaptive Control (MRAC) scheme in the hovering flight phase. The adaptive laws are developed for the UAV system under nonparametric uncertainty (gust and wind disturbance). Lyapunov-based stability analysis of the adaptive controlled UAV system under roll motion has been conducted and the adaptive laws have been accordingly developed. The Uniform Ultimate Boundness (UUB) of tracking error has been proven and the stability analysis showed that the incorporation of dead-zone modification in adaptive laws could guarantee the uniform boundness of all signals. The computer simulation has been conducted based on a proposed controller for tracking control of the roll motion. The results show that the drift, which appears in estimated gain behaviors due to the application of gust and wind disturbance, could be stopped by introducing dead-zone modification in adaptive laws, which leads to better robustness characteristics of the adaptive controller.

The Remote Control of the Artillery Rocket Set as a Strongly Nonlinear System Subject to Random Loads

On the modern battlefield, fighting capabilities, such as speed, target detection range, target identification capabilities, and shooting effectiveness, of short-range artillery rocket sets (ARSs) are constantly being improved. Problems arise when attempting to successfully fire such kits in the face of disruption from both the cannon and the moving platform on which the cannon is mounted. Furthermore, the set is a variable mass system since it can fire anywhere from a few to dozens or even hundreds of missiles in a brief period of time, implying that the ARS is a highly nonlinear system of variable parameters (non-stationary). This work shows how to control such a system. If the ARS is placed on a moving basis where there is both a system and measurement noise, the state variables must be restored, and the ARS data must be filtered. Therefore, in addition to the LQR regulator, an extended Kalman filter was used. As a consequence of this synthesis, an LQG (linear quadratic Gaussian) regulator of ARS was obtained, which was used to follow the target along the line of sight. The key goal of this paper is to develop control algorithms that will increase the performance of ARS control in elevation and azimuth, as well as the accuracy of achieving and eliminating maneuverable air targets. Moreover, through the quality criterion adopted, we hope to affect control energy costs while maintaining control precision. Graphical representations of certain computational simulation results are provided.