Using a theory of mind to find best responses to memory-one strategies

Evolution of reciprocity with limited payoff memory

Direct reciprocity is a mechanism for the evolution of cooperation in repeated social interactions. According to the literature, individuals naturally learn to adopt conditionally cooperative strategies if they have multiple encounters with their partner. Corresponding models have greatly facilitated our understanding of cooperation, yet they often make strong assumptions on how individuals remember and process payoff information. For example, when strategies are updated through social learning, it is commonly assumed that individuals compare their average payoffs. This would require them to compute (or remember) their payoffs against everyone else in the population. To understand how more realistic constraints influence direct reciprocity, we consider the evolution of conditional behaviours when individuals learn based on more recent experiences. Even in the most extreme case that they only take into account their very last interaction, we find that cooperation can still evolve. However, such individuals adopt less generous strategies, and they cooperate less often than in the classical setup with average payoffs. Interestingly, once individuals remember the payoffs of two or three recent interactions, cooperation rates quickly approach the classical limit. These findings contribute to a literature that explores which kind of cognitive capabilities are required for reciprocal cooperation. While our results suggest that some rudimentary form of payoff memory is necessary, it suffices to remember a few interactions.

Adaptive dynamics of memory-one strategies in the repeated donation game

Human interactions can take the form of social dilemmas: collectively, people fare best if all cooperate but each individual is tempted to free ride. Social dilemmas can be resolved when individuals interact repeatedly. Repetition allows them to adopt reciprocal strategies which incentivize cooperation. The most basic model for direct reciprocity is the repeated donation game, a variant of the prisoner's dilemma. Two players interact over many rounds; in each round they decide whether to cooperate or to defect. Strategies take into account the history of the play. Memory-one strategies depend only on the previous round. Even though they are among the most elementary strategies of direct reciprocity, their evolutionary dynamics has been difficult to study analytically. As a result, much previous work has relied on simulations. Here, we derive and analyze their adaptive dynamics. We show that the four-dimensional space of memory-one strategies has an invariant three-dimensional subspace, generated by the memory-one counting strategies. Counting strategies record how many players cooperated in the previous round, without considering who cooperated. We give a partial characterization of adaptive dynamics for memory-one strategies and a full characterization for memory-one counting strategies.

Mutation enhances cooperation in direct reciprocity

Direct reciprocity is a powerful mechanism for the evolution of cooperation based on repeated interactions between the same individuals. But high levels of cooperation evolve only if the benefit-to-cost ratio exceeds a certain threshold that depends on memory length. For the best-explored case of one-round memory, that threshold is two. Here, we report that intermediate mutation rates lead to high levels of cooperation, even if the benefit-to-cost ratio is only marginally above one, and even if individuals only use a minimum of past information. This surprising observation is caused by two effects. First, mutation generates diversity which undermines the evolutionary stability of defectors. Second, mutation leads to diverse communities of cooperators that are more resilient than homogeneous ones. This finding is relevant because many real-world opportunities for cooperation have small benefit-to-cost ratios, which are between one and two, and we describe how direct reciprocity can attain cooperation in such settings. Our result can be interpreted as showing that diversity, rather than uniformity, promotes evolution of cooperation.

Evolution of cooperation through cumulative reciprocity

Reciprocity is a simple principle for cooperation that explains many of the patterns of how humans seek and receive help from each other. To capture reciprocity, traditional models often assume that individuals use simple strategies with restricted memory. These memory-1 strategies are mathematically convenient, but they miss important aspects of human reciprocity, where defections can have lasting effects. Here we instead propose a strategy of cumulative reciprocity. Cumulative reciprocators count the imbalance of cooperation across their previous interactions with their opponent. They cooperate as long as this imbalance is sufficiently small. Using analytical and computational methods, we show that this strategy can sustain cooperation in the presence of errors, that it enforces fair outcomes and that it evolves in hostile environments. Using an economic experiment, we confirm that cumulative reciprocity is more predictive of human behaviour than several classical strategies. The basic principle of cumulative reciprocity is versatile and can be extended to a range of social dilemmas.

Direct reciprocity between individuals that use different strategy spaces

In repeated interactions, players can use strategies that respond to the outcome of previous rounds. Much of the existing literature on direct reciprocity assumes that all competing individuals use the same strategy space. Here, we study both learning and evolutionary dynamics of players that differ in the strategy space they explore. We focus on the infinitely repeated donation game and compare three natural strategy spaces: memory-1 strategies, which consider the last moves of both players, reactive strategies, which respond to the last move of the co-player, and unconditional strategies. These three strategy spaces differ in the memory capacity that is needed. We compute the long term average payoff that is achieved in a pairwise learning process. We find that smaller strategy spaces can dominate larger ones. For weak selection, unconditional players dominate both reactive and memory-1 players. For intermediate selection, reactive players dominate memory-1 players. Only for strong selection and low cost-to-benefit ratio, memory-1 players dominate the others. We observe that the supergame between strategy spaces can be a social dilemma: maximum payoff is achieved if both players explore a larger strategy space, but smaller strategy spaces dominate.

Misperception influence on zero-determinant strategies in iterated Prisoner's Dilemma

Zero-determinant (ZD) strategies have attracted wide attention in Iterated Prisoner's Dilemma (IPD) games, since the player equipped with ZD strategies can unilaterally enforce the two players' expected utilities subjected to a linear relation. On the other hand, uncertainties, which may be caused by misperception, occur in IPD inevitably in practical circumstances. To better understand the situation, we consider the influence of misperception on ZD strategies in IPD, where the two players, player X and player Y, have different cognitions, but player X detects the misperception and it is believed to make ZD strategies by player Y. We provide a necessary and sufficient condition for the ZD strategies in IPD with misperception, where there is also a linear relationship between players' utilities in player X's cognition. Then we explore bounds of players' expected utility deviation from a linear relationship in player X's cognition with also improving its own utility.

Cooperation in alternating interactions with memory constraints

In repeated social interactions, individuals often employ reciprocal strategies to maintain cooperation. To explore the emergence of reciprocity, many theoretical models assume synchronized decision making. In each round, individuals decide simultaneously whether to cooperate or not. Yet many manifestations of reciprocity in nature are asynchronous. Individuals provide help at one time and receive help at another. Here, we explore such alternating games in which players take turns. We mathematically characterize all Nash equilibria among memory-one strategies. Moreover, we use evolutionary simulations to explore various model extensions, exploring the effect of discounted games, irregular alternation patterns, and higher memory. In all cases, we observe that mutual cooperation still evolves for a wide range of parameter values. However, compared to simultaneous games, alternating games require different strategies to maintain cooperation in noisy environments. Moreover, none of the respective strategies are evolutionarily stable.

In many instances of reciprocity, individuals cooperate in turns. Here, the authors analyze the equilibria and the dynamics of such alternating games, and in particular describe all strategies with one-round memory that maintain cooperation.

Human players manage to extort more than the mutual cooperation payoff in repeated social dilemmas

Social dilemmas are mixed-motive games. Although the players have a common interest in maintaining cooperation, each may try to obtain a larger payoff by cooperating less than the other. This phenomenon received increased attention after Press and Dyson discovered a class of strategies for the repeated prisoner's dilemma (extortionate strategies) that secure for themselves a payoff that is never smaller, but can be larger, than the opponent's payoff. We conducted an experiment to test whether humans adopt extortionate strategies when playing a social dilemma. Our results reveal that human subjects do try to extort a larger payoff from their opponents. However, they are only successful when extortionate strategies are part of a Nash equilibrium. In settings where extortionate strategies do not appear in any Nash equilibrium, attempts at extortion only result in a breakdown of cooperation. Our subjects recognized the different incentives implied by the two settings, and they were ready to "extort" the opponent when allowed to do so. This suggests that deviations from mutually cooperative equilibria, which are usually attributed to players' impatience, coordination problems, or lack of information, can instead be driven by subjects trying to reach more favorable outcomes.