Adaptive Reinforcement Learning Control Based on Neural Approximation for Nonlinear Discrete-Time Systems With Unknown Nonaffine Dead-Zone Input

Optimized Adaptive Fuzzy Security Control of Nonlinear Systems With Prescribed Tracking Performance

This article studies the optimized fuzzy prescribed performance control problem for nonlinear nonstrict-feedback systems under denial-of-service (DoS) attacks. A fuzzy estimator is delicately designed to model the immeasurable system states in the presence of DoS attacks. To achieve the preset tracking performance, a simper prescribed performance error transformation is constructed considering the characteristics of DoS attacks, which helps obtain a novel Hamilton-Jacobi-Bellman equation to derive the optimized prescribed performance controller. Furthermore, the fuzzy-logic system, combined with the reinforcement learning (RL) technique, is employed to approximate the unknown nonlinearity existing in the prescribed performance controller design process. An optimized adaptive fuzzy security control law is then proposed for the considered nonlinear nonstrict-feedback systems subject to DoS attacks. Through the Lyapunov stability analysis, the tracking error is proved to approach the predefined region by the preset finite time, even in the presence of DoS attacks. Meanwhile, the consumed control resources are minimized due to the RL-based optimized algorithm. Finally, an actual example with comparisons verifies the effectiveness of the proposed control algorithm.

Intelligent Control of Flexible Hypersonic Flight Dynamics With Input Dead Zone Using Singular Perturbation Decomposition

This article studies the robust intelligent control for the longitudinal dynamics of flexible hypersonic flight vehicle with input dead zone. Considering the different time-scale characteristics among the system states, the singular perturbation decomposition is employed to transform the rigid-elastic coupling model into the slow dynamics and the fast dynamics. For the slow dynamics with unknown system nonlinearities, the robust neural control is constructed using the switching mechanism to achieve the coordination between robust design and neural learning. For the time-varying control gain caused by unknown dead-zone input, the stable control is presented with an adaptive estimation design. For the fast dynamics, the sliding mode control is constructed to make the elastic modes stable and convergent. The elevator deflection is obtained by combining the two control signals. The stability of the dynamics is analyzed through the Lyapunov approach and the system tracking errors are bounded. The simulation is conducted to demonstrate the effectiveness of the proposed approach.

Fault-tolerant control for nonlinear systems with a dead zone: Reinforcement learning approach

This paper focuses on the adaptive reinforcement learning-based optimal control problem for standard nonstrict-feedback nonlinear systems with the actuator fault and an unknown dead zone. To simultaneously reduce the computational complexity and eliminate the local optimal problem, a novel neural network weight updated algorithm is presented to replace the classic gradient descent method. By utilizing the backstepping technique, the actor critic-based reinforcement learning control strategy is developed for high-order nonlinear nonstrict-feedback systems. In addition, two auxiliary parameters are presented to deal with the input dead zone and actuator fault respectively. All signals in the system are proven to be semi-globally uniformly ultimately bounded by Lyapunov theory analysis. At the end of the paper, some simulation results are shown to illustrate the remarkable effect of the proposed approach.

Explicit Linear Left-and-Right 5-Step Formulas With Zeroing Neural Network for Time-Varying Applications

In this article, being different from conventional time-discretization (simply called discretization) formulas, explicit linear left-and-right 5-step (ELLR5S) formulas with sixth-order precision are proposed. The general sixth-order ELLR5S formula with four variable parameters is developed first, and constraints of these four parameters are displayed to guarantee the zero stability, consistence, and convergence of the formula. Then, by choosing specific parameter values within constraints, eight specific sixth-order ELLR5S formulas are developed. The general sixth-order ELLR5S formula is further utilized to generate discrete zeroing neural network (DZNN) models for solving time-varying linear and nonlinear systems. For comparison, three conventional discretization formulas are also utilized. Theoretical analyses are presented to show the performance of ELLR5S formulas and DZNN models. Furthermore, abundant experiments, including three practical applications, that is, angle-of-arrival (AoA) localization and two redundant manipulators (PUMA560 manipulator and Kinova manipulator) control, are conducted. The synthesized results substantiate the efficacy and superiority of sixth-order ELLR5S formulas as well as the corresponding DZNN models.

Adaptive Multigradient Recursive Reinforcement Learning Event-Triggered Tracking Control for Multiagent Systems

This article proposes a fault-tolerant adaptive multigradient recursive reinforcement learning (RL) event-triggered tracking control scheme for strict-feedback discrete-time multiagent systems. The multigradient recursive RL algorithm is used to avoid the local optimal problem that may exist in the gradient descent scheme. Different from the existing event-triggered control results, a new lemma about the relative threshold event-triggered control strategy is proposed to handle the compensation error, which can improve the utilization of communication resources and weaken the negative impact on tracking accuracy and closed-loop system stability. To overcome the difficulty caused by sensor fault, a distributed control method is introduced by adopting the adaptive compensation technique, which can effectively decrease the number of online estimation parameters. Furthermore, by using the multigradient recursive RL algorithm with less learning parameters, the online estimation time can be effectively reduced. The stability of closed-loop multiagent systems is proved by using the Lyapunov stability theorem, and it is verified that all signals are semiglobally uniformly ultimately bounded. Finally, two simulation examples are given to show the availability of the presented control scheme.

Indefinite Mean-Field Stochastic Cooperative Linear-Quadratic Dynamic Difference Game With Its Application to the Network Security Model

In this article, we show how to obtain all of the Pareto optimal decision vectors and solutions for the finite horizon indefinite mean-field stochastic cooperative linear-quadratic (LQ) difference game. First, the equivalence between the solvability of the introduced N coupled generalized difference Riccati equations (GDREs) and the solvability of the multiobjective optimization problem is established. However, it is difficult to obtain Pareto optimal decision vectors based on the N coupled GDREs because the optimal joint strategy adopted by all players to optimize the performance criterion of some players in the game is different from the strategies of other players, which rely on the weighted matrices of cost functionals that may be different among players. Second, a necessary and sufficient condition is developed to guarantee the convexity of the costs, which makes the weighting technique not only sufficient but also necessary for searching Pareto optimal decision vectors. It is then shown that the mean-field Pareto optimality algorithm (MF-POA) is presented to identify, in principle, all of the Pareto optimal decision vectors and solutions via the solutions to the weighted coupled GDREs and the weighted coupled generalized difference Lyapunov equations (GDLEs), respectively. Finally, a cooperative network security game is reported to illustrate the results presented. Simulation results validate the solvability, correctness, and efficiency of the proposed algorithm.

Target Tracking Control of a Biomimetic Underwater Vehicle Through Deep Reinforcement Learning

In this article, the underwater target tracking control problem of a biomimetic underwater vehicle (BUV) is addressed. Since it is difficult to build an effective mathematic model of a BUV due to the uncertainty of hydrodynamics, target tracking control is converted into the Markov decision process and is further achieved via deep reinforcement learning. The system state and reward function of underwater target tracking control are described. Based on the actor-critic reinforcement learning framework, the deep deterministic policy gradient actor-critic algorithm with supervision controller is proposed. The training tricks, including prioritized experience replay, actor network indirect supervision training, target network updating with different periods, and expansion of exploration space by applying random noise, are presented. Indirect supervision training is designed to address the issues of low stability and slow convergence of reinforcement learning in the continuous state and action space. Comparative simulations are performed to show the effectiveness of the training tricks. Finally, the proposed actor-critic reinforcement learning algorithm with supervision controller is applied to the physical BUV. Swimming pool experiments of underwater object tracking of the BUV are conducted in multiple scenarios to verify the effectiveness and robustness of the proposed method.

Adaptive Neural Network Control for Full-State Constrained Robotic Manipulator With Actuator Saturation and Time-Varying Delays

This article proposes an adaptive neural network (NN) control method for an n -link constrained robotic manipulator. Driven by actual demands, manipulator and actuator dynamics, state and input constraints, and unknown time-varying delays are taken into account simultaneously. NNs are employed to approximate unknown nonlinearities. Time-varying barrier Lyapunov functions are utilized to cope with full-state constraints. By resorting to saturation function and Lyapunov-Krasovskii functionals, the effects of actuator saturation and time delays are eliminated. It is proved that all the closed-loop signals are semiglobally uniformly ultimately bounded, full-state constraints and actuator saturation are not violated, and error signals remain within compact sets around zero. Simulation studies are given to demonstrate the validity and advantages of this control scheme.

Actor-critic learning based coordinated control for a dual-arm robot with prescribed performance and unknown backlash-like hysteresis

In this paper, we focus on the tracking problem of a dual-arm robot (DAR) with prescribed performance and unknown input backlash-like hysteresis. Considering this problem, adaptive coordinated control with actor-critic (AC) design is proposed. Motivated by the increasing control requirements, prescribed performance is imposed on the DAR system to guarantee the tracking performance. In order to improve the self-learning ability and handle the problems caused by the input backlash-like hysteresis and system uncertainty, AC learning (ACL) algorithm is introduced. Through the cost function about tracking errors, a critic network is adopted to judge the control performance. An actor network is adopted to obtain the control input based on the critic result, where the system uncertainty and unknown part of the input backlash-like hysteresis are approximated by neural networks (NNs). In addition, the system stability is proven by the Lyapunov direct method. Numerical simulation is finally conducted to further testify the validity of the proposed coordinated control with AC design for the DAR system.

Adaptive Fuzzy Finite-Time Tracking Control for Nonstrict Full States Constrained Nonlinear System With Coupled Dead-Zone Input

This article proposes an adaptive finite-time tracking control based on fuzzy-logic systems (FLSs) for an uncertain nonstrict nonlinear multi-input-multi-output (MIMO) full-state-constrained system with the coupled uncertain dead-zone input. By using three kinds of FLSs: the uncertain system, the uncertain dead zone, and the uncertain input transfer inverse matrix are approximated using the system function FLS, dead-zone FLS, and input transfer inverse matrix FLS, respectively. After defining the barrier Lyapunov function, the fuzzy-based adaptive tracking controllers are designed, and the fuzzy weights are updated through the proposed adaptive laws. Then, based on the extended finite-time convergence theorem, with the design parameters chosen properly, the target uncertain nonlinear system is guaranteed to be semiglobal practical finite-time stable (SGPFS); and the full-state constraints are not violated while avoiding the effects of the dead zones. Furthermore, a simulation is presented to verify the validity of the proposed algorithm.

Solver-Critic: A Reinforcement Learning Method for Discrete-Time-Constrained-Input Systems

In this article, a solver-critic (SC) architecture is developed for optimal control problems of discrete-time (DT)-constrained-input systems. The proposed design consists of three parts: 1) a critic network; 2) an action solver; and 3) a target network. The critic network first approximates the action-value function using the sum-of-squares (SOS) polynomial. Then, the action solver adopts the SOS programming to obtain control inputs within the constraint set. The target network introduces the soft update mechanism into policy evaluation to stabilize the learning process. By using the proposed architecture, the constrained-input control problem can be solved without adding the nonquadratic functionals into the reward function. In this article, the theoretical analysis of the convergence property is presented. Besides, the effects of both different initial Q -functions and different discount factors are investigated. It is proven that the learned policy converges to the optimal solution of the Hamilton-Jacobi-Bellman equation. Four numerical examples are provided to validate the theoretical analysis and also demonstrate the effectiveness of our approach.

Artificial Intelligence Methodologies for Data Management

This study analyses the main challenges, trends, technological approaches, and artificial intelligence methods developed by new researchers and professionals in the field of machine learning, with an emphasis on the most outstanding and relevant works to date. This literature review evaluates the main methodological contributions of artificial intelligence through machine learning. The methodology used to study the documents was content analysis; the basic terminology of the study corresponds to machine learning, artificial intelligence, and big data between the years 2017 and 2021. For this study, we selected 181 references, of which 120 are part of the literature review. The conceptual framework includes 12 categories, four groups, and eight subgroups. The study of data management using AI methodologies presents symmetry in the four machine learning groups: supervised learning, unsupervised learning, semi-supervised learning, and reinforced learning. Furthermore, the artificial intelligence methods with more symmetry in all groups are artificial neural networks, Support Vector Machines, K-means, and Bayesian Methods. Finally, five research avenues are presented to improve the prediction of machine learning.

Neuroadaptive Fault-Tolerant Control Under Multiple Objective Constraints With Applications to Tire Production Systems

Many manufacturing systems not only involve nonlinearities and nonvanishing disturbances but also are subject to actuation failures and multiple yet possibly conflicting objectives, making the underlying control problem interesting and challenging. In this article, we present a neuroadaptive fault-tolerant control solution capable of addressing those factors concurrently. To cope with the multiple objective constraints, we propose a method to accommodate these multiple objectives in such a way that they are all confined in certain range, distinguishing itself from the traditional method that seeks for a common optimum (which might not even exist due to the complicated and conflicting objective requirement) for all the objective functions. By introducing a novel barrier function, we convert the system under multiple constraints into one without constraints, allowing for the nonconstrained control algorithms to be derived accordingly. The system uncertainties and the unknown actuation failures are dealt with by using the deep-rooted information-based method. Furthermore, by utilizing a transformed signal as the initial filter input, we integrate dynamic surface control (DSC) into backstepping design to eliminate the feasibility conditions completely and avoid off-line parameter optimization. It is shown that, with the proposed neuroadaptive control scheme, not only stable system operation is maintained but also each objective function is confined within the prespecified region, which could be asymmetric and time-varying. The effectiveness of the algorithm is validated via simulation on speed regulation of extruding machine in tire production lines.

Inverse-Free Discrete ZNN Models Solving for Future Matrix Pseudoinverse via Combination of Extrapolation and ZeaD Formulas

Time-varying matrix pseudoinverse (TVMP) problem has been investigated by many researchers in recent years, but a new class of matrix termed Zhang matrix has been found and not been handled by some conventional models, e.g., Getz-Marsden dynamic model. On the other way, future matrix pseudoinverse (FMP), as a more challenging and intractable discrete-time problem, deserves more attention due to its significant role-playing on some engineering applications, such as redundant manipulator. Based on the zeroing neural network (ZNN), this article concentrates on designing new discrete ZNN models appropriately for computing the FMPs of all matrices of full rank, including the Zhang matrix. First, an inverse-free continuous ZNN model for computing TVMP is derived. Subsequently, Zhang et al. discretization (ZeaD) formulas and equidistant extrapolation formulas are used to discretize the continuous ZNN model to two discrete ZNN models for computing FMPs with different truncation errors. The numerical experiments are conducted for the five conventional discrete models and two new discrete ZNN models. Distinct numerical results substantiate the effectiveness and choiceness of newly proposed models. Finally, one of the newly proposed models is implemented on simulating and physical instances of robot manipulators, respectively, to show its practicability.

A Novel Supertwisting Zeroing Neural Network With Application to Mobile Robot Manipulators

Various zeroing neural network (ZNN) models have been investigated to address the tracking control of robot manipulators for the capacity of parallel processing and nonlinearity handling. However, two limitations occur in the existing ZNN models. The first one is the convergence time that tends to be infinitely large. The second one is the research of robustness that remains in the analyses of stability and asymptotic convergence. To simultaneously enhance the convergence performance and robustness, this article proposes a new ZNN model by using a supertwisting (ST) algorithm, termed STZNN model, for the tracking control of mobile robot manipulators. The proposed STZNN model inherently possesses the advantages of finite-time convergence and robustness making the control process fast and robust. The bridge from the sliding mode control to the ZNN is built, and the essential connection between the ST algorithm and ZNN is explored by constructing a unified design process. Theorems and proofs about global stability, finite-time convergence, and robustness are provided. Finally, path-tracking applications, comparisons, and tests substantiate the effectiveness and superiority of the STZNN model for the tracking control handling of mobile robot manipulators.

Adaptive Neural Network-Based Finite-Time Online Optimal Tracking Control of the Nonlinear System With Dead Zone

Considering the uncertain nonstrict nonlinear system with dead-zone input, an adaptive neural network (NN)-based finite-time online optimal tracking control algorithm is proposed. By using the tracking errors and the Lipschitz linearized desired tracking function as the new state vector, an extended system is present. Then, a novel Hamilton-Jacobi-Bellman (HJB) function is defined to associate with the nonquadratic performance function. Further, the upper limit of integration is selected as the finite-time convergence time, in which the dead-zone input is considered. In addition, the Bellman error function can be obtained from the Hamiltonian function. Then, the adaptations of the critic and action NN are updated by using the gradient descent method on the Bellman error function. The semiglobal practical finite-time stability (SGPFS) is guaranteed, and the tracking errors convergence to a compact set by zero in a finite time.

Quantized controller for a class of uncertain nonlinear systems with dead-zone nonlinearity

In this paper, a quantized controller is designed for a class of uncertain nonlinear systems subjected to unknown disturbances and unknown dead-zone nonlinearity. A general class of strict feedback nonlinear systems is taken as the plant to design the controller. Here, each differential equation of the system is considered to have unknown parameters and time-varying disturbances. The maximum upper bound of the disturbances is estimated instead of estimating each disturbance. This novel idea reduces the computational cost in handling the disturbances in uncertain systems. The tuning functions are constructed to estimate the unknown system parameter and maximum upper bound of the disturbances. It is considered that the actuator dead-zone nonlinearity is bounded by an unknown parameter and incorporated to design the final quantized controller. A backstepping technique is applied to design the tuning functions and controller that stabilizes the uncertain system. The stability of the proposed controller is proved using the Lyapunov stability based theory. The obtained MATLAB simulation test results verify the designed proposed controller.

Adaptive Neural Quantized Control of MIMO Nonlinear Systems Under Actuation Faults and Time-Varying Output Constraints

In this article, a neural network (NN)-based robust adaptive fault-tolerant control (FTC) algorithm is proposed for a class of multi-input multi-output (MIMO) strict-feedback nonlinear systems with input quantization and actuation faults as well as asymmetric yet time-varying output constraints. By introducing a key nonlinear decomposition for quantized input, the developed control scheme does not require the detailed information of quantization parameters. By imposing a reasonable condition on the gain matrix under actuation faults, together with the inherent approximation capability of NN, the difficulty of FTC design caused by anomaly actuation can be handled gracefully, and the normally used yet rigorous assumption on control gain matrix in most existing results is significantly relaxed. Furthermore, a brand new barrier function is constructed to handle the asymmetric yet time-varying output constraints such that the analysis and design are extremely simplified compared with the traditional barrier Lyapunov function (BLF)-based methods. NNs are used to approximate the unknown nonlinear continuous functions. The stability of the closed-loop system is analyzed by using the Lyapunov method and is verified through a simulation example.

New Super-Twisting Zeroing Neural-Dynamics Model for Tracking Control of Parallel Robots: A Finite-Time and Robust Solution

Parallel robots are usually required to perform real-time tracking control tasks in the presence of external disturbances in the complex environment. Conventional zeroing neural-dynamics (ZNDs) provide an alternative solution for the real-time tracking control of parallel robots due to its capacity of parallel processing and nonlinearity handling. However, it is still a challenge for the solution in a unified framework of the ZND to deal with the external disturbances, and simultaneously possess a finite-time convergence property. In this paper, a novel ZND model by exploring the super-twisting (ST) algorithm, named ST-ZND model, is proposed. The theoretical analyses on the global stability, finite-time convergence, as well as the robustness against the external disturbances are rigorously presented. Finally, the effectiveness and superiority of the ST-ZND model for the real-time tracking control of parallel robots are demonstrated by two illustrative examples, comparisons, and convergence tests.

Composite Neural Learning-Based Nonsingular Terminal Sliding Mode Control of MEMS Gyroscopes

The efficient driving control of MEMS gyroscopes is an attractive way to improve the precision without hardware redesign. This paper investigates the sliding mode control (SMC) for the dynamics of MEMS gyroscopes using neural networks (NNs). Considering the existence of the dynamics uncertainty, the composite neural learning is constructed to obtain higher tracking precision using the serial-parallel estimation model (SPEM). Furthermore, the nonsingular terminal SMC (NTSMC) is proposed to achieve finite-time convergence. To obtain the prescribed performance, a time-varying barrier Lyapunov function (BLF) is introduced to the control scheme. Through simulation tests, it is observed that under the BLF-based NTSMC with composite learning design, the tracking precision of MEMS gyroscopes is highly improved.

Adaptive Optimal Control for a Class of Nonlinear Systems: The Online Policy Iteration Approach

This paper studies the online adaptive optimal controller design for a class of nonlinear systems through a novel policy iteration (PI) algorithm. By using the technique of neural network linear differential inclusion (LDI) to linearize the nonlinear terms in each iteration, the optimal law for controller design can be solved through the relevant algebraic Riccati equation (ARE) without using the system internal parameters. Based on PI approach, the adaptive optimal control algorithm is developed with the online linearization and the two-step iteration, i.e., policy evaluation and policy improvement. The convergence of the proposed PI algorithm is also proved. Finally, two numerical examples are given to illustrate the effectiveness and applicability of the proposed method.

Adaptive Neural Network Learning Controller Design for a Class of Nonlinear Systems With Time-Varying State Constraints

This paper studies an adaptive neural network (NN) tracking control method for a class of uncertain nonlinear strict-feedback systems with time-varying full-state constraints. As we all know, the states are inevitably constrained in the actual systems because of the safety and performance factors. The main contributions of this paper are that: 1) in order to ensure that the states do not violate the asymmetric time-varying constraint regions, an adaptive NN controller is constructed by introducing the asymmetric time-varying barrier Lyapunov function (TVBLF) and 2) the amount of the learning parameters is reduced by introducing a TVBLF at each step of the backstepping. Based on the Lyapunov stability analysis, it can be proven that all the signals in the closed-loop system are the semiglobal ultimately uniformly bounded and the time-varying full-state constraints are never violated. Finally, a numerical simulation is given, and the effectiveness of this adaptive control method can be verified.