A Multiagent Reinforcement Learning Algorithm with Non-linear Dynamics

State transition learning with limited data for safe control of switched nonlinear systems

Switching dynamics are prevalent in real-world systems, arising from either intrinsic changes or responses to external influences, which can be appropriately modeled by switched systems. Control synthesis for switched systems, especially integrating safety constraints, is recognized as a significant and challenging topic. This study focuses on devising a learning-based control strategy for switched nonlinear systems operating under arbitrary switching law. It aims to maintain stability and uphold safety constraints despite limited system data. To achieve these goals, we employ the control barrier function method and Lyapunov theory to synthesize a controller that delivers both safety and stability performance. To overcome the difficulties associated with constructing the specific control barrier and Lyapunov function and take advantage of switching characteristics, we create a neural control barrier function and a neural Lyapunov function separately for control policies through a state transition learning approach. These neural barrier and Lyapunov functions facilitate the design of the safe controller. The corresponding control policy is governed by learning from two components: policy loss and forward state estimation. The effectiveness of the developing scheme is verified through simulation examples.

A Collaborative Multiagent Reinforcement Learning Method Based on Policy Gradient Potential

Gradient-based method has been extensively used in today's multiagent reinforcement learning (MARL). In a gradient-based MARL algorithm, each agent updates its parameterized strategy in the direction of the gradient of some performance index. However, studies on the convergence of the existing gradient-based MARL algorithms for identical interest games are quite few. In this article, we propose a policy gradient potential (PGP) algorithm that takes PGP as the source of information for guiding the strategy update, as opposed to the gradient itself, to learn the optimal joint strategy that has a maximal global reward. Since the payoff matrix and the joint strategy are often unavailable to the learning agents in reality, we consider the probability of obtaining the maximal reward as the performance index. Theoretical analysis of the PGP algorithm on the continuous model involving an identical interest repeated game shows that if the component action of every optimal joint action is unique, the critical points corresponding to all optimal joint actions are asymptotically stable. The PGP algorithm is experimentally studied and compared against other MARL algorithms on two commonly used collaborative tasks-the robots leaving a room task and the distributed sensor network task, as well as a real-world minefield navigation problem where only local state and local reward information are available. The results show that the PGP algorithm outperforms the other algorithms in terms of the cumulative reward and the number of time steps used in an episode.

Mining Dynamics: Using Data Mining Techniques to Analyze Multi-agent Learning

Analyzing the learning dynamics in multi-agent systems (MASs) has received growing attention in recent years. Theoretical analysis of the dynamics was only possible in simple domains and simple algorithms. When one or more of these restrictions do not apply, theoretical analysis becomes prohibitively difficult, and researchers rely on experimental analysis instead. In experimental analysis, researchers have used some global performance metric(s) as a rough approximation to the internal dynamics of the adaptive MAS. For example, if the overall payoff improved over time and eventually appeared to stabilize, then the learning dynamics were assumed to be stable as well. In this paper, we promote a middle ground between the thorough theoretical analysis and the high-level experimental analysis. We introduce the concept of mining dynamics and propose data-mining-based methodologies to analyze multi-agent learning dynamics. Using our methodologies, researchers can identify clusters of learning parameter values that lead to similar performance, and discover frequent sequences in agent dynamics. We verify the potential of our approach using the well-known iterated prisoner’s dilemma (with multiple states) domain.

Dynamically analyzing cell interactions in biological environments using multiagent social learning framework

BACKGROUND

Biological environment is uncertain and its dynamic is similar to the multiagent environment, thus the research results of the multiagent system area can provide valuable insights to the understanding of biology and are of great significance for the study of biology. Learning in a multiagent environment is highly dynamic since the environment is not stationary anymore and each agent's behavior changes adaptively in response to other coexisting learners, and vice versa. The dynamics becomes more unpredictable when we move from fixed-agent interaction environments to multiagent social learning framework. Analytical understanding of the underlying dynamics is important and challenging.

RESULTS

In this work, we present a social learning framework with homogeneous learners (e.g., Policy Hill Climbing (PHC) learners), and model the behavior of players in the social learning framework as a hybrid dynamical system. By analyzing the dynamical system, we obtain some conditions about convergence or non-convergence. We experimentally verify the predictive power of our model using a number of representative games. Experimental results confirm the theoretical analysis.

CONCLUSION

Under multiagent social learning framework, we modeled the behavior of agent in biologic environment, and theoretically analyzed the dynamics of the model. We present some sufficient conditions about convergence or non-convergence and prove them theoretically. It can be used to predict the convergence of the system.

Independent reinforcement learners in cooperative Markov games: a survey regarding coordination problems

In the framework of fully cooperative multi-agent systems, independent (non-communicative) agents that learn by reinforcement must overcome several difficulties to manage to coordinate. This paper identifies several challenges responsible for the non-coordination of independent agents: Pareto-selection, non-stationarity, stochasticity, alter-exploration and shadowed equilibria. A selection of multi-agent domains is classified according to those challenges: matrix games, Boutilier's coordination game, predators pursuit domains and a special multi-state game. Moreover, the performance of a range of algorithms for independent reinforcement learners is evaluated empirically. Those algorithms are Q-learning variants: decentralized Q-learning, distributed Q-learning, hysteretic Q-learning, recursive frequency maximum Q-value and win-or-learn fast policy hill climbing. An overview of the learning algorithms’ strengths and weaknesses against each challenge concludes the paper and can serve as a basis for choosing the appropriate algorithm for a new domain. Furthermore, the distilled challenges may assist in the design of new learning algorithms that overcome these problems and achieve higher performance in multi-agent applications.

Multiagent reinforcement learning: spiking and nonspiking agents in the iterated Prisoner's Dilemma

This paper investigates multiagent reinforcement learning (MARL) in a general-sum game where the payoffs' structure is such that the agents are required to exploit each other in a way that benefits all agents. The contradictory nature of these games makes their study in multiagent systems quite challenging. In particular, we investigate MARL with spiking and nonspiking agents in the Iterated Prisoner's Dilemma by exploring the conditions required to enhance its cooperative outcome. The spiking agents are neural networks with leaky integrate-and-fire neurons trained with two different learning algorithms: 1) reinforcement of stochastic synaptic transmission, or 2) reward-modulated spike-timing-dependent plasticity with eligibility trace. The nonspiking agents use a tabular representation and are trained with Q- and SARSA learning algorithms, with a novel reward transformation process also being applied to the Q-learning agents. According to the results, the cooperative outcome is enhanced by: 1) transformed internal reinforcement signals and a combination of a high learning rate and a low discount factor with an appropriate exploration schedule in the case of non-spiking agents, and 2) having longer eligibility trace time constant in the case of spiking agents. Moreover, it is shown that spiking and nonspiking agents have similar behavior and therefore they can equally well be used in a multiagent interaction setting. For training the spiking agents in the case where more than one output neuron competes for reinforcement, a novel and necessary modification that enhances competition is applied to the two learning algorithms utilized, in order to avoid a possible synaptic saturation. This is done by administering to the networks additional global reinforcement signals for every spike of the output neurons that were not "responsible" for the preceding decision.