Adaptive critic designs for optimal control of uncertain nonlinear systems with unmatched interconnections

Modeling the effect of magnetoelectric nanoparticles on neuronal electrical activity: An analog circuit approach

This paper introduces a physical neuron model that incorporates magnetoelectric nanoparticles (MENPs) as an essential electrical circuit component to wirelessly control local neural activity. Availability of such a model is important as MENPs, due to their magnetoelectric effect, can wirelessly and noninvasively modulate neural activity, which, in turn, has implications for both finding cures for neurological diseases and creating a wireless noninvasive high-resolution brain-machine interface. When placed on a neuronal membrane, MENPs act as magnetic-field-controlled finite-size electric dipoles that generate local electric fields across the membrane in response to magnetic fields, thus allowing to controllably activate local ion channels and locally initiate an action potential. Herein, the neuronal electrical characteristic description is based on ion channel activation and inhibition mechanisms. A MENP-based memristive Hodgkin-Huxley circuit model is extracted by combining the Hodgkin-Huxley model and an equivalent circuit model for a single MENP. In this model, each MENP becomes an integral part of the neuron, thus enabling wireless local control of the neuron's electric circuit itself. Furthermore, the model is expanded to include multiple MENPs to describe collective effects in neural systems.

Reinforcement learning for robust stabilization of nonlinear systems with asymmetric saturating actuators

We study the robust stabilization problem of a class of nonlinear systems with asymmetric saturating actuators and mismatched disturbances. Initially, we convert such a robust stabilization problem into a nonlinear-constrained optimal control problem by constructing a discounted cost function for the auxiliary system. Then, for the purpose of solving the nonlinear-constrained optimal control problem, we develop a simultaneous policy iteration (PI) in the reinforcement learning framework. The implementation of the simultaneous PI relies on an actor-critic architecture, which employs actor and critic neural networks (NNs) to separately approximate the control policy and the value function. To determine the actor and critic NNs' weights, we use the approach of weighted residuals together with the typical Monte-Carlo integration technique. Finally, we perform simulations of two nonlinear plants to validate the established theoretical claims.

Decentralized Neurocontroller Design With Critic Learning for Nonlinear-Interconnected Systems

We consider the decentralized control problem of a class of continuous-time nonlinear systems with mismatched interconnections. Initially, with the discounted cost functions being introduced to auxiliary subsystems, we have the decentralized control problem converted into a set of optimal control problems. To derive solutions to these optimal control problems, we first present the related Hamilton-Jacobi-Bellman equations (HJBEs). Then, we develop a novel critic learning method to solve these HJBEs. To implement the newly developed critic learning approach, we only use critic neural networks (NNs) and tune their weight vectors via the combination of a modified gradient descent method and concurrent learning. By using the present critic learning method, we not only remove the restriction of initial admissible control but also relax the persistence-of-excitation condition. After that, we employ Lyapunov's direct method to demonstrate that the critic NNs' weight estimation error and the states of closed-loop auxiliary systems are stable in the sense of uniform ultimate boundedness. Finally, we separately provide a nonlinear-interconnected plant and an unstable interconnected power system to validate the present critic learning approach.

Decentralized Event-Driven Constrained Control Using Adaptive Critic Designs

We study the decentralized event-driven control problem of nonlinear dynamical systems with mismatched interconnections and asymmetric input constraints. To begin with, by introducing a discounted cost function for each auxiliary subsystem, we transform the decentralized event-driven constrained control problem into a group of nonlinear H₂ -constrained optimal control problems. Then, we develop the event-driven Hamilton-Jacobi-Bellman equations (ED-HJBEs), which arise in the nonlinear H₂ -constrained optimal control problems. Meanwhile, we demonstrate that all the solutions of the ED-HJBEs together keep the overall system stable in the sense of uniform ultimate boundedness (UUB). To solve the ED-HJBEs, we build a critic-only architecture under the framework of adaptive critic designs. The architecture only employs critic neural networks and updates their weight vectors via the gradient descent method. After that, based on the Lyapunov approach, we prove that the UUB stability of all signals in the closed-loop auxiliary subsystems is assured. Finally, simulations of an illustrated nonlinear interconnected plant are provided to validate the present designs.

Adaptive-Critic Design for Decentralized Event-Triggered Control of Constrained Nonlinear Interconnected Systems Within an Identifier-Critic Framework

This article studies the decentralized event-triggered control problem for a class of constrained nonlinear interconnected systems. By assigning a specific cost function for each constrained auxiliary subsystem, the original control problem is equivalently transformed into finding a series of optimal control policies updating in an aperiodic manner, and these optimal event-triggered control laws together constitute the desired decentralized controller. It is strictly proven that the system under consideration is stable in the sense of uniformly ultimate boundedness provided by the solutions of event-triggered Hamilton-Jacobi-Bellman equations. Different from the traditional adaptive critic design methods, we present an identifier-critic network architecture to relax the restrictions posed on the system dynamics, and the actor network commonly used to approximate the optimal control law is circumvented. The weights in the critic network are tuned on the basis of the gradient descent approach as well as the historical data, such that the persistence of excitation condition is no longer needed. The validity of our control scheme is demonstrated through a simulation example.

A Trajectory Optimization Strategy for Connected and Automated Vehicles at Junction of Freeway and Urban Road

The phenomenon of stop-and-go traffic and its environmental impact has become a crucial issue that needs to be tackled, in terms of the junctions between freeway and urban road networks, which consist of freeway off-ramps, downstream intersections, and the junction section. The development of Connected and Automated Vehicles (CAVs) has provided promising solutions to tackle the difficulties that arise along intersections and freeway off-ramps separately. However, several problems still exist that need to be handled in terms of junction structure, including vehicle merging trajectory optimization, vehicle crossing trajectory optimization, and heterogeneous decision-making. In this paper, a two-stage CAV trajectory optimization strategy is presented to improve fuel economy and to reduce delays through a joint framework. The first stage considers an approach to determine travel time considering the different topological structures of each subarea to ensure maximum capacity. In the second stage, Pontryagin’s Minimum Principle (PMP) is employed to construct Hamiltonian equations to smooth vehicle trajectory under the requirements of vehicle dynamics and safety. Targeted methods are devised to avoid driving backwards and to ensure an optimal vehicle gap, which make up for the shortcomings of the PMP theory. Finally, simulation experiments are designed to verify the effectiveness of the proposed strategy. The evaluation results show that our strategy could effectively militate travel delays and fuel consumption.

Approximate Dynamic Programming for Nonlinear-Constrained Optimizations

In this paper, we study the constrained optimization problem of a class of uncertain nonlinear interconnected systems. First, we prove that the solution of the constrained optimization problem can be obtained through solving an array of optimal control problems of constrained auxiliary subsystems. Then, under the framework of approximate dynamic programming, we present a simultaneous policy iteration (SPI) algorithm to solve the Hamilton-Jacobi-Bellman equations corresponding to the constrained auxiliary subsystems. By building an equivalence relationship, we demonstrate the convergence of the SPI algorithm. Meanwhile, we implement the SPI algorithm via an actor-critic structure, where actor networks are used to approximate optimal control policies and critic networks are applied to estimate optimal value functions. By using the least squares method and the Monte Carlo integration technique together, we are able to determine the weight vectors of actor and critic networks. Finally, we validate the developed control method through the simulation of a nonlinear interconnected plant.

Decentralized Event-Triggered Control for a Class of Nonlinear-Interconnected Systems Using Reinforcement Learning

In this article, we propose a novel decentralized event-triggered control (ETC) scheme for a class of continuous-time nonlinear systems with matched interconnections. The present interconnected systems differ from most of the existing interconnected plants in that their equilibrium points are no longer assumed to be zero. Initially, we establish a theorem to indicate that the decentralized ETC law for the overall system can be represented by an array of optimal ETC laws for nominal subsystems. Then, to obtain these optimal ETC laws, we develop a reinforcement learning (RL)-based method to solve the Hamilton-Jacobi-Bellman equations arising in the discounted-cost optimal ETC problems of the nominal subsystems. Meanwhile, we only use critic networks to implement the RL-based approach and tune the critic network weight vectors by using the gradient descent method and the concurrent learning technique together. With the proposed weight vectors tuning rule, we are able to not only relax the persistence of the excitation condition but also ensure the critic network weight vectors to be uniformly ultimately bounded. Moreover, by utilizing the Lyapunov method, we prove that the obtained decentralized ETC law can force the entire system to be stable in the sense of uniform ultimate boundedness. Finally, we validate the proposed decentralized ETC strategy through simulations of the nonlinear-interconnected systems derived from two inverted pendulums connected via a spring.

Adaptive Critic Learning for Constrained Optimal Event-Triggered Control With Discounted Cost

This article studies an optimal event-triggered control (ETC) problem of nonlinear continuous-time systems subject to asymmetric control constraints. The present nonlinear plant differs from many studied systems in that its equilibrium point is nonzero. First, we introduce a discounted cost for such a system in order to obtain the optimal ETC without making coordinate transformations. Then, we present an event-triggered Hamilton-Jacobi-Bellman equation (ET-HJBE) arising in the discounted-cost constrained optimal ETC problem. After that, we propose an event-triggering condition guaranteeing a positive lower bound for the minimal intersample time. To solve the ET-HJBE, we construct a critic network under the framework of adaptive critic learning. The critic network weight vector is tuned through a modified gradient descent method, which simultaneously uses historical and instantaneous state data. By employing the Lyapunov method, we prove that the uniform ultimate boundedness of all signals in the closed-loop system is guaranteed. Finally, we provide simulations of a pendulum system and an oscillator system to validate the obtained optimal ETC strategy.

Particle swarm optimized neural networks based local tracking control scheme of unknown nonlinear interconnected systems

In this paper, a local tracking control (LTC) scheme is developed via particle swarm optimized neural networks (PSONN) for unknown nonlinear interconnected systems. With the local input-output data, a local neural network identifier is constructed to approximate the local input gain matrix and the mismatched interconnection, which are utilized to derive the LTC. To solve the local Hamilton-Jacobi-Bellman equation, a local critic NN is established to estimate the proper local value function, which reflects the mismatched interconnection. The weight vector of the local critic NN is trained online by particle swarm optimization, thus the success rate of system execution is increased. The stability of the closed-loop unknown nonlinear interconnected system is guaranteed to be uniformly ultimately bounded through Lyapunov's direct method. Simulation results of two examples demonstrate the effectiveness of the developed PSONN-based LTC scheme.

Integral reinforcement learning based event-triggered control with input saturation

In this paper, a novel integral reinforcement learning (IRL)-based event-triggered adaptive dynamic programming scheme is developed for input-saturated continuous-time nonlinear systems. By using the IRL technique, the learning system does not require the knowledge of the drift dynamics. Then, a single critic neural network is designed to approximate the unknown value function and its learning is not subjected to the requirement of an initial admissible control. In order to reduce computational and communication costs, the event-triggered control law is designed. The triggering threshold is given to guarantee the asymptotic stability of the control system. Two examples are employed in the simulation studies, and the results verify the effectiveness of the developed IRL-based event-triggered control method.