Neural-network-based synchronous iteration learning method for multi-player zero-sum games

Nearly Optimal Control for Mixed Zero-Sum Game Based on Off-Policy Integral Reinforcement Learning

In this article, we solve a class of mixed zero-sum game with unknown dynamic information of nonlinear system. A policy iterative algorithm that adopts integral reinforcement learning (IRL), which does not depend on system information, is proposed to obtain the optimal control of competitor and collaborators. An adaptive update law that combines critic-actor structure with experience replay is proposed. The actor function not only approximates optimal control of every player but also estimates auxiliary control, which does not participate in the actual control process and only exists in theory. The parameters of the actor-critic structure are simultaneously updated. Then, it is proven that the parameter errors of the polynomial approximation are uniformly ultimately bounded. Finally, the effectiveness of the proposed algorithm is verified by two given simulations.

Observer-based event-triggered control for zero-sum games of input constrained multi-player nonlinear systems

In this paper, an event-triggered control (ETC) method is investigated to solve zero-sum game (ZSG) problems of unknown multi-player continuous-time nonlinear systems with input constraints by using adaptive dynamic programming (ADP). To relax the requirement of system dynamics, a neural network (NN) observer is constructed to identify the dynamics of multi-player system via the input and output data. Then, the event-triggered Hamilton-Jacobi-Isaacs (HJI) equation of the ZSG can be solved by constructing a critic NN, and the approximated optimal control law and the worst disturbance law can be obtained directly. A triggering scheme which determines the updating time instants of the control law and the disturbance law is developed. Thus, the proposed ADP-based ETC method cannot only reduce the computational burden, but also save communication resource and bandwidths. Furthermore, we prove that the signals of the closed-loop system and the approximate errors of the critic NN weights are uniformly ultimately bounded by using Lyapunov's direct method, and the Zeno behavior is excluded. Finally, two simulation examples are provided to demonstrate the effectiveness of the proposed ETC scheme.

Minmax fuzzy deterministic policy gradient for zero-sum differential game: Take pursuit-evasion problem as example

A novel actor-critic algorithm is introduced and applied to zero-sum differential game. The proposed novel structure consists of two actors and a critic. Different actors represent the control policies of different players, and the critic is used to approximate the state-action utility function. Instead of neural network, the fuzzy inference system is applied as approximators for the actors and critic so that the specific practical meaning can be represented by the linguistic fuzzy rules. Since the goals of the players in the game are completely opposite, the actors for different players are simultaneously updated in opposite directions during the training. One actor is updated updated toward the direction that can minimize the Q value while the other updated toward the direction that can maximize the Q value. A pursuit-evasion problem with two pursuers and one evader is taken as an example to illustrate the validity of our method. In this problem, the two pursuers the same actor and the symmetry in the problem is used to improve the replay buffer. At the end of this paper, some confrontations between the policies with different training episodes are conducted.

Distributed zero-sum differential game for multi-agent systems in strict-feedback form with input saturation and output constraint

This paper investigates the distributed differential game tracking problem for nonlinear multi-agent systems with output constraint under a fixed directed graph. Each follower can be taken as strict-feedback structure with uncertain nonlinearities and input saturation. Firstly, by utilizing the command filtered backstepping technique, the distributed tracking control problem of multi-agent systems in strict-feedback form can be transformed into an equivalent distributed differential game problem of tracking error dynamics in affine form by designing a distributed feedforward tracking controller, in which neural networks (NNs) and the auxiliary system are introduced to deal with the unknown nonlinearities and input saturation, respectively. Especially, a novel barrier Lyapunov function (BLF) is firstly introduced to tackle with the output constraint. Subsequently, by using adaptive dynamic programming (ADP) technique, the distributed zero-sum differential game strategy is derived, in which a critic network is constructed to approximate the cooperative cost function online with a novel updating law. Therefore, the whole distributed control scheme not only guarantees the closed-loop signals to be cooperatively uniformly ultimately bounded (CUUB), but also ensures the cooperative cost function to be minimized. Meanwhile, the output constraint and input saturation are not violated. Finally, simulation results demonstrate the effectiveness of the proposed method.