A model for the evolution of reinforcement learning in fluctuating games

Reinforcement learning with thermal fluctuations at the nanoscale

Reinforcement Learning offers a framework to learn to choose actions in order to control a system. However, at small scales Brownian fluctuations limit the control of nanomachine actuation or nanonavigation and of the molecular machinery of life. We analyze this regime using the general framework of Markov decision processes. We show that at the nanoscale, while optimal control actions should bring an improvement proportional to the small ratio of the applied force times a length scale over the temperature, the learned improvement is smaller and proportional to the square of this small ratio. Consequently, the efficiency of learning, which compares the learning improvement to the theoretical optimal improvement, drops to zero. Nevertheless, these limitations can be circumvented by using actions learned at a lower temperature. These results are illustrated with simulations of the control of the shape of small particle clusters.

A neural network model for the evolution of learning in changing environments

Learning from past experience is an important adaptation and theoretical models may help to understand its evolution. Many of the existing models study simple phenotypes and do not consider the mechanisms underlying learning while the more complex neural network models often make biologically unrealistic assumptions and rarely consider evolutionary questions. Here, we present a novel way of modelling learning using small neural networks and a simple, biology-inspired learning algorithm. Learning affects only part of the network, and it is governed by the difference between expectations and reality. We use this model to study the evolution of learning under various environmental conditions and different scenarios for the trade-off between exploration (learning) and exploitation (foraging). Efficient learning readily evolves in our individual-based simulations. However, in line with previous studies, the evolution of learning is less likely in relatively constant environments, where genetic adaptation alone can lead to efficient foraging, or in short-lived organisms that cannot afford to spend much of their lifetime on exploration. Once learning does evolve, the characteristics of the learning strategy (i.e. the duration of the learning period and the learning rate) and the average performance after learning are surprisingly little affected by the frequency and/or magnitude of environmental change. In contrast, an organism's lifespan and the distribution of resources in the environment have a clear effect on the evolved learning strategy: a shorter lifespan or a broader resource distribution lead to fewer learning episodes and larger learning rates. Interestingly, a longer learning period does not always lead to better performance, indicating that the evolved neural networks differ in the effectiveness of learning. Overall, however, we show that a biologically inspired, yet relatively simple, learning mechanism can evolve to lead to an efficient adaptation in a changing environment.

Hamilton's rule, the evolution of behavior rules and the wizardry of control theory

This paper formalizes selection on a quantitative trait affecting the evolution of behavior (or development) rules through which individuals act and react with their surroundings. Combining Hamilton's marginal rule for selection on scalar traits and concepts from optimal control theory, a necessary first-order condition for the evolutionary stability of the trait in a group-structured population is derived. The model, which is of intermediate level of complexity, fills a gap between the formalization of selection on evolving traits that are directly conceived as actions (no phenotypic plasticity) and selection on evolving traits that are conceived as strategies or function valued actions (complete phenotypic plasticity). By conceptualizing individuals as open deterministic dynamical systems expressing incomplete phenotypic plasticity, the model captures selection on a large class of phenotypic expression mechanisms, including developmental pathways and learning under life-history trade-offs. As an illustration of the results, a first-order condition for the evolutionary stability of behavior response rules from the social evolution literature is re-derived, strengthened, and generalized. All results of the paper also generalize directly to selection on multidimensional quantitative traits affecting behavior rule evolution, thereby covering neural and gene network evolution.

Resource preferences and the emergence of individual niche specialization within populations

Growing evidence that individuals of many generalist animals behave as resource specialists have attracted substantial research interest for its ecological and evolutionary implications. Variation in resource preferences is considered to be critical for developing a general theory of individual specialization. However, it remains to be shown whether diverging preferences can arise among individuals sharing a similar environment, and whether these preferences are sufficiently stable over time to be ecologically relevant. We addressed these issues by means of common garden experiments in feral pigeons (Columba livia), a species known to exhibit among-individual resource specialization in the wild. Food-choice experiments on wild-caught pigeons and their captive-bred cross-fostered descendants showed that short-term variation in food preferences can easily arise within a population, and that this variation may represent a substantial fraction of the population foraging niche. However, the experiments also showed that, rather than being limited by genetic or vertical cultural inheritance, food preferences exhibited high plasticity and tended to converge in the long-term. Although our results challenge the notion that variation in food preferences is a major driver of resource specialization, early differences in preferences could pave the way to specializations when combined with neophobic responses and/or positive feedbacks that reinforce niche conservation.

Modeling the evolution of sensitive periods

In the past decade, there has been monumental progress in our understanding of the neurobiological basis of sensitive periods. Little is known, however, about the evolution of sensitive periods. Recent studies have started to address this gap. Biologists have built mathematical models exploring the environmental conditions in which sensitive periods are likely to evolve. These models investigate how mechanisms of plasticity can respond optimally to experience during an individual's lifetime. This paper discusses the central tenets, insights, and predictions of these models, in relation to empirical work on humans and other animals. We also discuss which future models are needed to improve the bridge between theory and data, advancing their synergy. Our paper is written in an accessible manner and for a broad audience. We hope our work will contribute to recently emerging connections between the fields of developmental neuroscience and evolutionary biology.

Deconstructing Evolutionary Game Theory: Coevolution of Social Behaviors with Their Evolutionary Setting

Evolution of social behaviors is one of the most fascinating and active fields of evolutionary biology. During the past half century, social evolution theory developed into a mature field with powerful tools to understand the dynamics of social traits such as cooperation under a wide range of conditions. In this article, I argue that the next stage in the development of social evolution theory should consider the evolution of the setting in which social behaviors evolve. To that end, I propose a conceptual map of the components that make up the evolutionary setting of social behaviors, review existing work that considers the evolution of each component, and discuss potential future directions. The theoretical work reviewed here illustrates how unexpected dynamics can happen when the setting of social evolution itself is evolving, such as cooperation sometimes being self-limiting. I argue that a theory of how the setting of social evolution itself evolves will lead to a deeper understanding of when cooperation and other social behaviors evolve and diversify.

Enriching behavioral ecology with reinforcement learning methods

This article focuses on the division of labor between evolution and development in solving sequential, state-dependent decision problems. Currently, behavioral ecologists tend to use dynamic programming methods to study such problems. These methods are successful at predicting animal behavior in a variety of contexts. However, they depend on a distinct set of assumptions. Here, we argue that behavioral ecology will benefit from drawing more than it currently does on a complementary collection of tools, called reinforcement learning methods. These methods allow for the study of behavior in highly complex environments, which conventional dynamic programming methods do not feasibly address. In addition, reinforcement learning methods are well-suited to studying how biological mechanisms solve developmental and learning problems. For instance, we can use them to study simple rules that perform well in complex environments. Or to investigate under what conditions natural selection favors fixed, non-plastic traits (which do not vary across individuals), cue-driven-switch plasticity (innate instructions for adaptive behavioral development based on experience), or developmental selection (the incremental acquisition of adaptive behavior based on experience). If natural selection favors developmental selection, which includes learning from environmental feedback, we can also make predictions about the design of reward systems. Our paper is written in an accessible manner and for a broad audience, though we believe some novel insights can be drawn from our discussion. We hope our paper will help advance the emerging bridge connecting the fields of behavioral ecology and reinforcement learning.

Helping in humans and other animals: a fruitful interdisciplinary dialogue

Humans are arguably unique in the extent and scale of cooperation with unrelated individuals. While pairwise interactions among non-relatives occur in some non-human species, there is scant evidence of the large-scale, often unconditional prosociality that characterizes human social behaviour. Consequently, one may ask whether research on cooperation in humans can offer general insights to researchers working on similar questions in non-human species, and whether research on humans should be published in biology journals. We contend that the answer to both of these questions is yes. Most importantly, social behaviour in humans and other species operates under the same evolutionary framework. Moreover, we highlight how an open dialogue between different fields can inspire studies on humans and non-human species, leading to novel approaches and insights. Biology journals should encourage these discussions rather than drawing artificial boundaries between disciplines. Shared current and future challenges are to study helping in ecologically relevant contexts in order to correctly interpret how payoff matrices translate into inclusive fitness, and to integrate mechanisms into the hitherto largely functional theory. We can and should study human cooperation within a comparative framework in order to gain a full understanding of the evolution of helping.

Why mutual helping in most natural systems is neither conflict-free nor based on maximal conflict

Mutual helping for direct benefits can be explained by various game theoretical models, which differ mainly in terms of the underlying conflict of interest between two partners. Conflict is minimal if helping is self-serving and the partner benefits as a by-product. In contrast, conflict is maximal if partners are in a prisoner's dilemma with both having the pay-off-dominant option of not returning the other's investment. Here, we provide evolutionary and ecological arguments for why these two extremes are often unstable under natural conditions and propose that interactions with intermediate levels of conflict are frequent evolutionary endpoints. We argue that by-product helping is prone to becoming an asymmetric investment game since even small variation in by-product benefits will lead to the evolution of partner choice, leading to investments by the chosen class. Second, iterated prisoner's dilemmas tend to take place in stable social groups where the fitness of partners is interdependent, with the effect that a certain level of helping is self-serving. In sum, intermediate levels of mutual helping are expected in nature, while efficient partner monitoring may allow reaching higher levels.