Learning in a game context: strategy choice by some keeps learning from evolving in others

A neural network model for the evolution of learning in changing environments

Learning from past experience is an important adaptation and theoretical models may help to understand its evolution. Many of the existing models study simple phenotypes and do not consider the mechanisms underlying learning while the more complex neural network models often make biologically unrealistic assumptions and rarely consider evolutionary questions. Here, we present a novel way of modelling learning using small neural networks and a simple, biology-inspired learning algorithm. Learning affects only part of the network, and it is governed by the difference between expectations and reality. We use this model to study the evolution of learning under various environmental conditions and different scenarios for the trade-off between exploration (learning) and exploitation (foraging). Efficient learning readily evolves in our individual-based simulations. However, in line with previous studies, the evolution of learning is less likely in relatively constant environments, where genetic adaptation alone can lead to efficient foraging, or in short-lived organisms that cannot afford to spend much of their lifetime on exploration. Once learning does evolve, the characteristics of the learning strategy (i.e. the duration of the learning period and the learning rate) and the average performance after learning are surprisingly little affected by the frequency and/or magnitude of environmental change. In contrast, an organism's lifespan and the distribution of resources in the environment have a clear effect on the evolved learning strategy: a shorter lifespan or a broader resource distribution lead to fewer learning episodes and larger learning rates. Interestingly, a longer learning period does not always lead to better performance, indicating that the evolved neural networks differ in the effectiveness of learning. Overall, however, we show that a biologically inspired, yet relatively simple, learning mechanism can evolve to lead to an efficient adaptation in a changing environment.

The importance of distinguishing individual differences in 'social impact' versus 'social responsiveness' when quantifying indirect genetic effects on the evolution of social plasticity

Social evolution and the dynamics of social interactions have previously been studied under the frameworks of quantitative genetics and behavioural ecology. In quantitative genetics, indirect genetic effects of social partners on the socially plastic phenotypes of focal individuals typically lack crucial detail already included in treatments of social plasticity in behavioural ecology. Specifically, whilst focal individuals (e.g. receivers) may show variation in their 'responsiveness' to the social environment, individual social partners (e.g. signallers) may have a differential 'impact' on focal phenotypes. Here we propose an integrative framework, that highlights the distinction between responsiveness versus impact in indirect genetic effects for a range of behavioural traits. We describe impact and responsiveness using a reaction norm approach and provide statistical models for the assessment of these effects of focal and social partner identity in different types of social interactions. By providing such a framework, we hope to stimulate future quantitative research investigating the causes and consequences of social interactions on phenotypic evolution.

A reaction norm framework for the evolution of learning: how cumulative experience shapes phenotypic plasticity

Learning is a familiar process to most people, but it currently lacks a fully developed theoretical position within evolutionary biology. Learning (memory and forgetting) involves adjustments in behaviour in response to cumulative sequences of prior experiences or exposures to environmental cues. We therefore suggest that all forms of learning (and some similar biological phenomena in development, aging, acquired immunity and acclimation) can usefully be viewed as special cases of phenotypic plasticity, and formally modelled by expanding the concept of reaction norms to include additional environmental dimensions quantifying sequences of cumulative experience (learning) and the time delays between events (forgetting). Memory therefore represents just one of a number of different internal neurological, physiological, hormonal and anatomical ‘states’ that mediate the carry‐over effects of cumulative environmental experiences on phenotypes across different time periods. The mathematical and graphical conceptualisation of learning as plasticity within a reaction norm framework can easily accommodate a range of different ecological scenarios, closely linking statistical estimates with biological processes. Learning and non‐learning plasticity interact whenever cumulative prior experience causes a modification in the reaction norm (a) elevation [mean phenotype], (b) slope [responsiveness], (c) environmental estimate error [informational memory] and/or (d) phenotypic precision [skill acquisition]. Innovation and learning new contingencies in novel (laboratory) environments can also be accommodated within this approach. A common reaction norm approach should thus encourage productive cross‐fertilisation of ideas between traditional studies of learning and phenotypic plasticity. As an example, we model the evolution of plasticity with and without learning under different levels of environmental estimation error to show how learning works as a specific adaptation promoting phenotypic plasticity in temporally autocorrelated environments. Our reaction norm framework for learning and analogous biological processes provides a conceptual and mathematical structure aimed at usefully stimulating future theoretical and empirical investigations into the evolution of plasticity across a wider range of ecological contexts, while providing new interdisciplinary connections regarding learning mechanisms.

Slow-Fast Cognitive Phenotypes and Their Significance for Social Behavior: What Can We Learn From Honeybees?

Cognitive variation is proposed to be the fundamental underlying factor that drives behavioral variation, yet it is still to be fully integrated with the observed variation at other phenotypic levels that has recently been unified under the common pace-of-life framework. This cognitive and the resulting behavioral diversity is especially significant in the context of a social group, the performance of which is a collective outcome of this diversity. In this review, we argue about the utility of classifying cognitive traits along a slow-fast continuum in the larger context of the pace-of-life framework. Using Tinbergen’s explanatory framework for different levels of analyses and drawing from the large body of knowledge about honeybee behavior, we discuss the observed interindividual variation in cognitive traits and slow-fast cognitive phenotypes from an adaptive, evolutionary, mechanistic and developmental perspective. We discuss the challenges in this endeavor and suggest possible next steps in terms of methodological, statistical and theoretical approaches to move the field forward for an integrative understanding of how slow-fast cognitive differences, by influencing collective behavior, impact social evolution.

On the importance of accounting for alternative foraging tactics when assessing cognitive performance in wild animals

Research Highlight: Reichert, S., Morand-Ferron, J., Kulahci, I. G., Firth, J. A., Davidson, G. L., Crofts, S. J., & Quinn, J. L. (2021) Cognition and covariance in the producer-scrounger game. Journal of Animal Ecology, https://doi.org/10.1111/1365-2656.13551. When foraging in groups, individuals can either acquire their own resources, as producers, or profit from the work of other individuals, as scroungers. Individuals vary in how much they rely on one foraging tactic over the other. Yet, each of these foraging tactics presents unique cognitive challenges. Using a field experiment with a mixed-species flock of birds, Reichert et al. (2021) investigated how production learning (i.e. successfully feeding from their assigned rewarded feeder) and scrounging propensity (i.e. collecting food from a non-assigned feeders by following another individual) are related at an individual level, as well as the repeatability of both production learning and scrounging propensity. The authors show that overall, (a) individuals learned to scrounge, (b) individuals who rely more on scrounging took longer to learn their assigned feeder and (c) variation in each cognitive trait was mostly explained by individual behavioural flexibility rather than by consistent differences between individuals. Since learning was negatively correlated with the use of an alternative foraging tactic (i.e. scrounging), results of this study also suggest that individual choice of foraging tactics should be considered when evaluating cognitive abilities in wild animals.

Can the setup of a patch modulate finder's advantage?

According to the Rate Maximization Model the finder's advantage is an environmental factor modulating social foraging strategies. One factor that can influence the finder's advantage is the patches' setup. We analyzed the strategies of Wistar rats foraging in groups n = 4 in a 4 × 3 array of deposits where only four of them were baited (patches) with different configurations on each trial. Specifically, the goal was to assess whether the finder's advantage varied according to the distribution (Square vs. Zig-zag) and distance (Small vs. Large) between patches. Foraging responses were classified as production (seeking for food) or scrounging (tracking conspecifics) to calculate the frequency of use of each strategy in each group and estimate the induced finder's advantage in each situation. In patch-setups with short distances, production was more common and the finder's advantage higher. Results can be explained by the combined effect of a local enhancing and simultaneous patch exploitation on small setups.

Cognition and covariance in the producer-scrounger game

The producer-scrounger game is a key element of foraging ecology in many systems. Producing and scrounging typically covary negatively, but partitioning this covariance into contributions of individual plasticity and consistent between individual differences is key to understanding population-level consequences of foraging strategies. Furthermore, little is known about the role cognition plays in the producer-scrounger game. We investigated the role of cognition in these alternative foraging tactics in wild mixed-species flocks of great tits and blue tits, using a production learning task in which we measured individuals' speed of learning to visit the single feeder in an array that would provide them with a food reward. We also quantified the proportion of individuals' feeds that were scrounges ('proportion scrounged'); scrounging was possible if individuals visited immediately after a previous rewarded visitor. Three learning experiments-initial and two reversal learning-enabled us to estimate the repeatability and covariance of each foraging behaviour. First, we examined whether individuals learned to improve their scrounging success (i.e. whether they obtained food by scrounging when there was an opportunity to do so). Second, we quantified the repeatability of proportion scrounged, and asked whether proportion scrounged affected production learning speed among individuals. Third, we used multivariate analyses to partition within- and among-individual components of covariance between proportion scrounged and production learning speed. Individuals improved their scrounging success over time. Birds with a greater proportion scrounged took longer to learn their own rewarding feeder. Although multivariate analyses showed that covariance between proportion scrounged and learning speed was driven primarily by within-individual variation, that is, by behavioural plasticity, among-individual differences also played a role for blue tits. This is the first demonstration of a cognitive trait influencing producing and scrounging in the same wild system, highlighting the importance of cognition in the use of alternative resource acquisition tactics. The results of our covariance analyses suggest the potential for genetic differences in allocation to alternative foraging tactics, which are likely species- and system-dependent. They also point to the need to control for different foraging tactics when studying individual cognition in the wild.

Competition for resources can promote the divergence of social learning phenotypes

Social learning occurs when animals acquire knowledge or skills by observing or interacting with others and is the fundamental building block of culture. Within populations, some individuals use social learning more frequently than others, but why social learning phenotypes differ among individuals is poorly understood. We modelled the evolution of social learning frequency in a system where foragers compete for resources, and there are many different foraging options to learn about. Social learning phenotypes diverged when some options offered much better rewards than others and expected rewards changed moderately quickly over time. When options offered similar rewards or when rewards changed slowly, a single social learning phenotype evolved. This held for fixed and simple conditional social learning rules. Sufficiently complex conditional social learning rules prevented the divergence of social learning phenotypes under all conditions. Our results explain how competition can promote the divergence of social learning phenotypes.

Measuring and understanding individual differences in cognition

Individuals vary in their cognitive performance. While this variation forms the foundation of the study of human psychometrics, its broader importance is only recently being recognized. Explicitly acknowledging this individual variation found in both humans and non-human animals provides a novel opportunity to understand the mechanisms, development and evolution of cognition. The papers in this special issue highlight the growing emphasis on individual cognitive differences from fields as diverse as neurobiology, experimental psychology and evolutionary biology. Here, we synthesize this body of work. We consider the distinct challenges in quantifying individual differences in cognition and provide concrete methodological recommendations. In particular, future studies would benefit from using multiple task variants to ensure they target specific, clearly defined cognitive traits and from conducting repeated testing to assess individual consistency. We then consider how neural, genetic, developmental and behavioural factors may generate individual differences in cognition. Finally, we discuss the potential fitness consequences of individual cognitive variation and place these into an evolutionary framework with testable hypotheses. We intend for this special issue to stimulate researchers to position individual variation at the centre of the cognitive sciences.This article is part of the theme issue 'Causes and consequences of individual differences in cognitive abilities'.

Are some individuals generally more behaviorally plastic than others? An experiment with sailfin mollies

Individuals within the same population generally differ among each other not only in their behavioral traits but also in their level of behavioral plasticity (i.e., in their propensity to modify their behavior in response to changing conditions). If the proximate factors underlying individual differences in behavioral plasticity were the same for any measure of plasticity, as commonly assumed, one would expect plasticity to be repeatable across behaviors and contexts. However, this assumption remains largely untested. Here, we conducted an experiment with sailfin mollies (Poecilia latipinna) whose behavioral plasticity was estimated both as the change in their personality traits or mating behavior across a social gradient and using their performance on a reversal-learning task. We found that the correlations between pairwise measures of plasticity were weak and non-significant, thus indicating that the most plastic individuals were not the same in all the tests. This finding might arise because either individuals adjust the magnitude of their behavioral responses depending on the benefits of plasticity, and/or individuals expressing high behavioral plasticity in one context are limited by neural and/or physiological constraints in the amount of plasticity they can express in other contexts. Because the repeatability of behavioral plasticity may have important evolutionary consequences, additional studies are needed to assess the importance of trade-offs between conflicting selection pressures on the maintenance of intra-individual variation in behavioral plasticity.

Energy state affects exploratory behavior of tree sparrows in a group context under differential food-patch distributions

BACKGROUND

When facing a novel situation, animals can retreat or leave to avoid risks, but will miss potential resources and opportunities. Alternatively they may reduce environmental uncertainty by exploration, while risking no energy rewards and exposure to hazards, and use the information retrieved for subsequent decision making. When exploring, however, animals may adopt different tactics according to individual states.

RESULTS

We tested that energy states will affect exploratory behavior by experimenting with wild-caught untrained Eurasian tree sparrows (Passer montanus) in fasted or fed states exploring in a novel space with hidden food supply in different patch distribution patterns. Our data revealed that fasted sparrows risked being earlier explorers more often, initiated more exploratory bouts before patches were found, and stayed longer on the ground under both patch patterns. Fasted sparrows discovered more patches and consumed more food than fed sparrows in dispersed, but not necessary so in clumped, patch patterns; whereas fed birds also increased patch finding to a certain level in dispersed patterns. Sparrows of both energy states, however, did not differ in feeding rates in either patch pattern.

CONCLUSIONS

Exploratory behavior of tree sparrows is state-dependent, which supports our prediction that birds with an energy shortage will be risk-prone and explore more readily. Our study also indicates a game nature of tree sparrow exploratory behavior in a group context when explorers are in different energy states and are exposed to different patch distributions. Birds of lower energy state adopting an active exploring tactic may be favored by obtaining higher energy gains in dispersed patch patterns with lower patch richness. More satiated birds, however, achieved a similar feeding rate by lowered exposure time.

Human collective intelligence under dual exploration-exploitation dilemmas

The exploration-exploitation dilemma is a recurrent adaptive problem for humans as well as non-human animals. Given a fixed time/energy budget, every individual faces a fundamental trade-off between exploring for better resources and exploiting known resources to optimize overall performance under uncertainty. Colonies of eusocial insects are known to solve this dilemma successfully via evolved coordination mechanisms that function at the collective level. For humans and other non-eusocial species, however, this dilemma operates within individuals as well as between individuals, because group members may be motivated to take excessive advantage of others' exploratory findings through social learning. Thus, even though social learning can reduce collective exploration costs, the emergence of disproportionate “information scroungers” may severely undermine its potential benefits. We investigated experimentally whether social learning opportunities might improve the performance of human participants working on a “multi-armed bandit” problem in groups, where they could learn about each other's past choice behaviors. Results showed that, even though information scroungers emerged frequently in groups, social learning opportunities reduced total group exploration time while increasing harvesting from better options, and consequentially improved collective performance. Surprisingly, enriching social information by allowing participants to observe others' evaluations of chosen options (e.g., Amazon's 5-star rating system) in addition to choice-frequency information had a detrimental impact on performance compared to the simpler situation with only the choice-frequency information. These results indicate that humans groups can handle the fundamental “dual exploration-exploitation dilemmas” successfully, and that social learning about simple choice-frequencies can help produce collective intelligence.

Evolution of learning strategies in temporally and spatially variable environments: a review of theory

The theoretical literature from 1985 to the present on the evolution of learning strategies in variable environments is reviewed, with the focus on deterministic dynamical models that are amenable to local stability analysis, and on deterministic models yielding evolutionarily stable strategies. Individual learning, unbiased and biased social learning, mixed learning, and learning schedules are considered. A rapidly changing environment or frequent migration in a spatially heterogeneous environment favors individual learning over unbiased social learning. However, results are not so straightforward in the context of learning schedules or when biases in social learning are introduced. The three major methods of modeling temporal environmental change--coevolutionary, two-timescale, and information decay--are compared and shown to sometimes yield contradictory results. The so-called Rogers' paradox is inherent in the two-timescale method as originally applied to the evolution of pure strategies, but is often eliminated when the other methods are used. Moreover, Rogers' paradox is not observed for the mixed learning strategies and learning schedules that we review. We believe that further theoretical work is necessary on learning schedules and biased social learning, based on models that are logically consistent and empirically pertinent.

Metabolic rates can drive individual differences in information and insurance use under the risk of starvation

Variation in how individuals invest in acquiring information (sampling) and in insuring themselves against potential negative consequences of uncertainty (e.g., by storing energy reserves) has been suggested to underlie consistent individual differences in suites of behavioral traits. However, the key drivers of individual differences in information use remain poorly understood. We use dynamic programming to explore how existing variation in metabolic rates (MRs) affects the use of sampling and insurance under starvation risk. Our analysis reveals nonlinear effects of MRs on diurnal patterns of sampling and insurance. Individuals with low MRs accrue reserves quickly, because they invest in sampling and are able to exploit profitable options when they arise. Individuals with intermediate MRs initially lose reserves, because sampling, while optimal, is relatively expensive; however, they later build reserves due to efficient exploitation of alternative foraging options. Sampling rarely pays for individuals with the highest MRs, which show relatively constant levels of energy reserves throughout the foraging period. Thus, individual variation in MRs on the scale observed in natural populations can lead to important differences in investment in sampling and insurance and may underpin consistent individual differences in suites of other behavioral traits, including individual differences in behavioral responsiveness.

The role of beginner's luck in learning to prefer risky patches by socially foraging house sparrows

Although there has been extensive research on the evolution of individual decision making under risk (when facing variable outcomes), little is known on how the evolution of such decision-making mechanisms has been shaped by social learning and exploitation. We presented socially foraging house sparrows with a choice between scattered feeding wells in which millet seeds were hidden under 2 types of colored sand: green sand offering ~80 seeds with a probability of 0.1 (high risk-high reward) and yellow sand offering 1 seed with certainty (low risk-low reward). Although the expected benefit of choosing variable wells was 8 times higher than that of choosing constant wells, only some sparrows developed a preference for variable wells, whereas others developed a significant preference for constant wells. We found that this dichotomy could be explained by stochastic individual differences in sampling success during foraging, rather than by social foraging strategies (active searching vs. joining others). Moreover, preference for variable or constant wells was related to the sparrows' success during searching, rather than during joining others or when picking exposed seeds (i.e., they learn when actively searching in the sand). Finally, although for many sparrows learning resulted in an apparently maladaptive risk aversion, group living still allowed them to enjoy profitable variable wells by occasionally joining variable-preferring sparrows.

Competition avoidance drives individual differences in response to a changing food resource in sticklebacks

Within the same population, individuals often differ in how they respond to changes in their environment. A recent series of models predicts that competition in a heterogeneous environment might promote between-individual variation in behavioural plasticity. We tested groups of sticklebacks in patchy foraging environments that differed in the level of competition. We also tested the same individuals across two different social groups and while alone to determine the social environment's influence on behavioural plasticity. In support of model predictions, individuals consistently differed in behavioural plasticity when the presence of conspecifics influenced the potential payoffs of a foraging opportunity. Whether individuals maintained their level of behavioural plasticity when placed in a new social group depended on the environmental heterogeneity. By explicitly testing predictions of recent theoretical models, we provide evidence for the types of ecological conditions under which we would expect, and not expect, variation in behavioural plasticity to be favoured.

LEARNING TO CHOOSE AMONG SOCIAL FORAGING STRATEGIES IN ADULT HOUSE SPARROWS (Passer domesticus)

Social foragers may be regarded as being engaged in a producer-scrounger game in which they can search for food independently or join others who have discovered food. Research on the producer-scrounger game has focused mainly on the different factors influencing its ESS solution, but very little is known about the actual mechanisms that shape players' decisions. Recent work has shown that early experience can affect producer-scrounger foraging tendencies in young house sparrows, and that in nutmeg mannikins learning is involved in reaching the ESS. Here we show that direct manipulation of the success rate experienced by adult sparrows when following others can change their strategy choice on the following day. We presented to live sparrows an experimental regime, where stuffed adult house sparrows in a feeding position were positioned on a foraging grid that included two reward regimes: a positive one, in which the stuffed models were placed near food, and a negative one, in which the models were placed away from food. There was a significant increase in joining behavior after the positive treatment (exhibited by 84% of the birds), but no change after the negative treatment. Further analysis demonstrated that sparrows more frequently used the strategy with which they were more successful (usually joining), and that differences in strategy use were correlated with differences in success. These results suggest that adult birds can monitor their success and learn to choose among social foraging strategies in the producer-scrounger game.

On the evolution of personalities via frequency-dependent selection

Personality differences can be found in a wide range of species across the animal kingdom, but why natural selection gave rise to such differences remains an open question. Frequency-dependent selection is a potent mechanism explaining variation; it does not explain, however, the other two key features associated with personalities, consistency and correlations. Using the hawk-dove game and a frequency-dependent foraging game as examples, we here show that this changes fundamentally whenever one takes into account the physiological architecture underlying behavior (e.g., metabolism). We find that the inclusion of physiology changes the evolutionary predictions concerning consistency and correlations: while selection gives rise to inconsistent individuals and stochastically fluctuating behavioral correlations in scenarios that neglect physiology, we find high levels of behavioral consistency and tight and stable trait correlations in scenarios that incorporate physiology. The coevolution of behavioral and physiological traits also gives rise to adaptive physiological differences that are systematically associated with behavioral differences. As well as providing a framework for understanding behavioral consistency and behavioral correlations, our work thus also provides an explanation for systematic physiological differences within populations, a phenomenon that appears to exist in a wide range of species but that, up to now, has been poorly understood.

Frequency-dependent payoffs and sequential decision-making favour consistent tactic use

Although natural selection should have favoured individuals capable of adjusting the weight they give to personal and social information according to circumstances, individuals generally differ consistently in their individual weighting of both types of information. Such individual differences are correlated with personality traits, suggesting that personality could directly affect individuals' ability to collect personal or social information. Alternatively, the link between personality and information use could simply emerge as a by-product of the sequential decision-making process in a frequency-dependent context. Indeed, when the gains associated with behavioural options depend on the choices of others, an individual's sequence of arrival could constrain its choice of options leading to the emergence of correlated behaviours. Any factor such as personality that affects decision order could thus be correlated with information use. To test this new explanation, we developed an individual-based model that simulates a group of animals engaged in a game of sequential frequency-dependent decision: a producer-scrounger game. Our results confirm that the sequence of decision, in this case enforced by the order in which animals enter a foraging area, consistently influences their mean tactic use and their individual plasticity, an outcome reminiscent of the correlation reported between personality and social information use.

Evolution of learned strategy choice in a frequency-dependent game

In frequency-dependent games, strategy choice may be innate or learned. While experimental evidence in the producer-scrounger game suggests that learned strategy choice may be common, a recent theoretical analysis demonstrated that learning by only some individuals prevents learning from evolving in others. Here, however, we model learning explicitly, and demonstrate that learning can easily evolve in the whole population. We used an agent-based evolutionary simulation of the producer-scrounger game to test the success of two general learning rules for strategy choice. We found that learning was eventually acquired by all individuals under a sufficient degree of environmental fluctuation, and when players were phenotypically asymmetric. In the absence of sufficient environmental change or phenotypic asymmetries, the correct target for learning seems to be confounded by game dynamics, and innate strategy choice is likely to be fixed in the population. The results demonstrate that under biologically plausible conditions, learning can easily evolve in the whole population and that phenotypic asymmetry is important for the evolution of learned strategy choice, especially in a stable or mildly changing environment.

Individual differences in plasticity and sampling when playing behavioural games

When engaged in behavioural games, animals can adjust their use of alternative tactics until groups reach stable equilibria. Recent theory on behavioural plasticity in games predicts that individuals should differ in their plasticity or responsiveness and hence in their degree of behavioural adjustment. Moreover, individuals are predicted to be consistent in their plasticity within and across biological contexts. These predictions have yet to be tested empirically and so we examine the behavioural adjustment of individual nutmeg mannikins (Lonchura punctulata), gregarious ground-feeding passerines, when playing two different social foraging games: producer-scrounger (PS) and patch-choice (PC) games. We found: (i) significant individual differences in plasticity and sampling behaviour in each of the two games, (ii) individual differences in sampling behaviour were consistent over different test conditions within a game (PC) and over a six month period (PS), (iii) but neither individual plasticity nor sampling behaviour was correlated from one social foraging game to another. The rate at which birds sampled alternative tactics was positively associated with seed intake in PS trials but negatively associated in PC trials. These results suggest that games with frequency dependence of pay-offs can maintain differences in behavioural plasticity but that an important component of this plasticity is group- and/or context-specific.