Reinforcement learning meets minority game: Toward optimal resource allocation

Catalytic evolution of cooperation in a population with behavioral bimodality

The remarkable adaptability of humans in response to complex environments is often demonstrated by the context-dependent adoption of different behavioral modes. However, the existing game-theoretic studies mostly focus on the single-mode assumption, and the impact of this behavioral multimodality on the evolution of cooperation remains largely unknown. Here, we study how cooperation evolves in a population with two behavioral modes. Specifically, we incorporate Q-learning and Tit-for-Tat (TFT) rules into our toy model and investigate the impact of the mode mixture on the evolution of cooperation. While players in a Q-learning mode aim to maximize their accumulated payoffs, players within a TFT mode repeat what their neighbors have done to them. In a structured mixing implementation where the updating rule is fixed for each individual, we find that the mode mixture greatly promotes the overall cooperation prevalence. The promotion is even more significant in the probabilistic mixing, where players randomly select one of the two rules at each step. Finally, this promotion is robust when players adaptively choose the two modes by a real-time comparison. In all three scenarios, players within the Q-learning mode act as catalyzers that turn the TFT players to be more cooperative and as a result drive the whole population to be highly cooperative. The analysis of Q-tables explains the underlying mechanism of cooperation promotion, which captures the "psychological evolution" in the players' minds. Our study indicates that the variety of behavioral modes is non-negligible and could be crucial to clarify the emergence of cooperation in the real world.

Emergence of cooperation under punishment: A reinforcement learning perspective

Punishment is a common tactic to sustain cooperation and has been extensively studied for a long time. While most of previous game-theoretic work adopt the imitation learning framework where players imitate the strategies of those who are better off, the learning logic in the real world is often much more complex. In this work, we turn to the reinforcement learning paradigm, where individuals make their decisions based upon their experience and long-term returns. Specifically, we investigate the prisoners' dilemma game with a Q-learning algorithm, and cooperators probabilistically pose punishment on defectors in their neighborhood. Unexpectedly, we find that punishment could lead to either continuous or discontinuous cooperation phase transitions, and the nucleation process of cooperation clusters is reminiscent of the liquid-gas transition. The analysis of a Q-table reveals the evolution of the underlying "psychologic" changes, which explains the nucleation process and different levels of cooperation. The uncovered first-order phase transition indicates that great care needs to be taken when implementing the punishment compared to the continuous scenario.

Evolutionary dynamics in financial markets with heterogeneities in investment strategies and reference points

In nature and human societies, the effects of homogeneous and heterogeneous characteristics on the evolution of collective behaviors are quite different from each other. By incorporating pair pattern strategies and reference point strategies into an agent-based model, we have investigated the effects of homogeneous and heterogeneous investment strategies and reference points on price movement. In the market flooded with the investors with homogeneous investment strategies or homogeneous reference points, large price fluctuations occur. In the market flooded with the investors with heterogeneous investment strategies or heterogeneous reference points, moderate price fluctuations occur. The coexistence of different kinds of investment strategies can not only refrain from the occurrence of large price fluctuations but also the occurrence of no-trading states. The present model reveals that the coexistence of heterogeneous populations, whether they are the individuals with heterogeneous investment strategies or heterogeneous reference points of stock prices, is an important factor for the stability of the stock market.

Market impact shapes competitive advantage of investment strategies in financial markets

The formation of an efficient market depends on the competition between different investment strategies, which accelerates all available information into asset prices. By incorporating market impact and two kinds of investment strategies into an agent-based model, we have investigated the coevolutionary mechanism of different investment strategies and the role of market impact in shaping a competitive advantage in financial markets. The coevolution of history-dependent strategies and reference point strategies depends on the levels of market impact and risk tolerance. For low market impact and low risk tolerance, the majority-win effect makes the trend-following strategies become dominant strategies. For high market impact and low risk tolerance, the minority-win effect makes the trend-rejecting strategies coupled with trend-following strategies become dominant strategies. The coupled effects of price fluctuations and strategy distributions have been investigated in depth. A U-shape distribution of history-dependent strategies is beneficial for a stable price, which is destroyed by the existence of reference point strategies with low risk tolerance. A δ-like distribution of history-dependent strategies leads to a large price fluctuation, which is suppressed by the existence of reference point strategies with high risk tolerance. The strategies that earn more in an inefficient market lose more in an efficient market. Such a result gives us another explanation for the principle of risk-profit equilibrium in financial markets: high return in an inefficient market should be coupled with high risk in an efficient market, low return in an inefficient market should be coupled with low risk in an efficient market.