Enriching behavioral ecology with reinforcement learning methods

Biological reinforcement learning simulation for natural enemy -host behavior: Exploring deep learning algorithms for population dynamics

This study introduces a simulation of biological reinforcement learning to explore the behavior of natural enemies in the presence of host pests, aiming to analyze the population dynamics between natural enemies and insect pests within an ecological context. The simulation leverages on Q-learning, a reinforcement learning algorithm, to model the decision-making processes of both parasitoids/predators and pests, thereby assessing the impact of varying parasitism and predation rates on pest population growth. Simulation parameters, such as episode count, duration in months, steps, learning rate, and discount factor, were set arbitrarily. Environmental and reward matrices, representing climatic conditions, crop availability, and the rewards for different actions, were established for each month. Initial Q-tables for parasitoids/predators and pests, along with population arrays, were used to track population dynamics.•The simulation, illustrated through the Aphid-Ladybird beetle interaction case study over multiple episodes, includes a sensitivity analysis to evaluate the effects of different predation rates.•Findings reveal detailed population dynamics, phase relationships between predator and pest populations, and the significant influence of predation rates.•These insights contribute to a deeper understanding of ecological systems and inform potential pest management strategies.

Behavioral selection in structured populations

The multilevel model of behavioral selection (MLBS) by Borgstede and Eggert (Behav Process 186:104370. 10.1016/j.beproc.2021.104370 , 2021) provides a formal framework that integrates reinforcement learning with natural selection using an extended Price equation. However, the MLBS is so far only formulated for homogeneous populations, thereby excluding all sources of variation between individuals. This limitation is of primary theoretical concern because any application of the MLBS to real data requires to account for variation between individuals. In this paper, I extend the MLBS to account for inter-individual variation by dividing the population into homogeneous sub-populations and including class-specific reproductive values as weighting factors for an individual's evolutionary fitness. The resulting formalism closes the gap between the theoretical underpinnings of behavioral selection and the application of the theory to empirical data, which naturally includes inter-individual variation. Furthermore, the extended MLBS is used to establish an explicit connection between the dynamics of learning and the maximization of individual fitness. These results expand the scope of the MLBS as a general theoretical framework for the quantitative analysis of learning and evolution.

The Uncontrollable Mortality Risk Hypothesis: Theoretical foundations and implications for public health

The 'Uncontrollable Mortality Risk Hypothesis' employs a behavioural ecological model of human health behaviours to explain the presence of social gradients in health. It states that those who are more likely to die due to factors beyond their control should be less motivated to invest in preventative health behaviours. We outline the theoretical assumptions of the hypothesis and stress the importance of incorporating evolutionary perspectives into public health. We explain how measuring perceived uncontrollable mortality risk can contribute towards understanding socioeconomic disparities in preventative health behaviours. We emphasize the importance of addressing structural inequalities in risk exposure, and argue that public health interventions should consider the relationship between overall levels of mortality risk and health behaviours across domains. We suggest that measuring perceptions of uncontrollable mortality risk can capture the unanticipated health benefits of structural risk interventions, as well as help to assess the appropriateness of different intervention approaches.

Specifying the timescale of early life unpredictability helps explain the development of internalising and externalising behaviours

Early life unpredictability is associated with both physical and mental health outcomes throughout the life course. Here, we classified adverse experiences based on the timescale on which they are likely to introduce variability in children's environments: variations unfolding over short time scales (e.g., hours, days, weeks) and labelled Stochasticity vs variations unfolding over longer time scales (e.g., months, years) and labelled Volatility and explored how they contribute to the development of problem behaviours. Results indicate that externalising behaviours at age 9 and 15 and internalising behaviours at age 15 were better accounted for by models that separated Stochasticity and Volatility measured at ages 3 to 5. Both externalising and internalising behaviours were specifically associated with Volatility, with larger effects for externalising behaviours. These findings are interpreted in light of evolutionary-developmental models of psychopathology and reinforcement learning models of learning under uncertainty.

Efficiency traps beyond the climate crisis: exploration-exploitation trade-offs and rebound effects

Higher levels of economic activity are often accompanied by higher energy use and consumption of natural resources. As fossil fuels still account for 80% of the global energy mix, energy consumption remains closely linked to greenhouse gas (GHG) emissions and thus to climate change. Under the assumption of sufficiently elastic demand, this reality of global economic development based on permanent growth of economic activity, brings into play the Jevons Paradox, which hypothesises that increases in the efficiency of resource use leads to increases in resource consumption. Previous research on the rebound effects has limitations, including a lack of studies on the connection between reinforcement learning and environmental consequences. This paper develops a mathematical model and computer simulator to study the effects of micro-level exploration-exploitation strategies on efficiency, consumption and sustainability, considering different levels of direct and indirect rebound effects. Our model shows how optimal exploration-exploitation strategies for increasing efficiency can lead to unsustainable development patterns if they are not accompanied by demand reduction measures, which are essential for mitigating climate change. Moreover, our paper speaks to the broader issue of efficiency traps by highlighting how indirect rebound effects not only affect primary energy (PE) consumption and GHG emissions, but also resource consumption in other domains. By linking these issues together, our study sheds light on the complexities and interdependencies involved in achieving sustainable development goals. This article is part of the theme issue 'Climate change adaptation needs a science of culture'.

What is the expected human childhood? Insights from evolutionary anthropology

In psychological research, there are often assumptions about the conditions that children expect to encounter during their development. These assumptions shape prevailing ideas about the experiences that children are capable of adjusting to, and whether their responses are viewed as impairments or adaptations. Specifically, the expected childhood is often depicted as nurturing and safe, and characterized by high levels of caregiver investment. Here, we synthesize evidence from history, anthropology, and primatology to challenge this view. We integrate the findings of systematic reviews, meta-analyses, and cross-cultural investigations on three forms of threat (infanticide, violent conflict, and predation) and three forms of deprivation (social, cognitive, and nutritional) that children have faced throughout human evolution. Our results show that mean levels of threat and deprivation were higher than is typical in industrialized societies, and that our species has experienced much variation in the levels of these adversities across space and time. These conditions likely favored a high degree of phenotypic plasticity, or the ability to tailor development to different conditions. This body of evidence has implications for recognizing developmental adaptations to adversity, for cultural variation in responses to adverse experiences, and for definitions of adversity and deprivation as deviation from the expected human childhood.

Suboptimal foraging theory: How inaccurate predictions and approximations can make better models of adaptive behavior

Optimal foraging theory (OFT) is based on the ecological concept that organisms select behaviors that convey future fitness, and on the mathematical concept of optimization: finding the alternative that provides the best value of a fitness measure. As implemented in, e.g., state-based dynamic modeling, OFT is powerful for one key problem of modern ecology: modeling behavior as a tradeoff among competing fitness elements such as growth, risk avoidance, and reproductive output. However, OFT is not useful for other modern problems such as representing feedbacks within systems of interacting, unique individuals: when we need to model foraging by each of many individuals that interact competitively or synergistically, optimization is impractical or impossible-there are no optimal behaviors. For such problems we can, however, still use the concept of future fitness to model behavior, by replacing optimization with less precise (but perhaps more realistic) techniques for ranking alternatives. Instead of simplifying the systems we model until we can find "optimal" behavior, we can use theory based on inaccurate predictions, coarse approximations, and updating to produce good behavior in more complex and realistic contexts. This "state- and prediction-based theory" (SPT) can, for example, produce realistic foraging decisions by each of many unique, interacting individuals when growth rates and predation risks vary over space and time. Because SPT lets us address more natural complexity and more realistic problems, it is more easily tested against more kinds of observation and more useful in management ecology. A simple foraging model illustrates how SPT readily accommodates complexities that make optimization intractable. Other models use SPT to represent contingent decisions (whether to feed or hide, in what patch) that are tradeoffs between growth and predation risk, when both growth and risk vary among hundreds of patches, vary unpredictably over time, depend on characteristics of the individuals, are subject to feedbacks from competition, and change over the daily light cycle. Modern ecology demands theory for tradeoff behaviors in complex contexts that produce feedbacks; when optimization is infeasible, we should not be afraid to use approximate fitness-seeking methods instead.

Sensitive periods, but not critical periods, evolve in a fluctuating environment: a model of incremental development

Sensitive periods, during which the impact of experience on phenotype is larger than in other periods, exist in all classes of organisms, yet little is known about their evolution. Recent mathematical modelling has explored the conditions in which natural selection favours sensitive periods. These models have assumed that the environment is stable across ontogeny or that organisms can develop phenotypes instantaneously at any age. Neither assumption generally holds. Here, we present a model in which organisms gradually tailor their phenotypes to an environment that fluctuates across ontogeny, while receiving cost-free, imperfect cues to the current environmental state. We vary the rate of environmental change, the reliability of cues and the duration of adulthood relative to ontogeny. We use stochastic dynamic programming to compute optimal policies. From these policies, we simulate levels of plasticity across ontogeny and obtain mature phenotypes. Our results show that sensitive periods can occur at the onset, midway through and even towards the end of ontogeny. In contrast with models assuming stable environments, organisms always retain residual plasticity late in ontogeny. We conclude that critical periods, after which plasticity is zero, are unlikely to be favoured in environments that fluctuate across ontogeny.

An evolutionary model of sensitive periods when the reliability of cues varies across ontogeny

Sensitive periods are widespread in nature, but their evolution is not well understood. Recent mathematical modeling has illuminated the conditions favoring the evolution of sensitive periods early in ontogeny. However, sensitive periods also exist at later stages of ontogeny, such as adolescence. Here, we present a mathematical model that explores the conditions that favor sensitive periods at later developmental stages. In our model, organisms use environmental cues to incrementally construct a phenotype that matches their environment. Unlike in previous models, the reliability of cues varies across ontogeny. We use stochastic dynamic programming to compute optimal policies for a range of evolutionary ecologies and then simulate developmental trajectories to obtain mature phenotypes. We measure changes in plasticity across ontogeny using study paradigms inspired by empirical research: adoption and cross-fostering. Our results show that sensitive periods only evolve later in ontogeny if the reliability of cues increases across ontogeny. The onset, duration, and offset of sensitive periods-and the magnitude of plasticity-depend on the specific parameter settings. If the reliability of cues decreases across ontogeny, sensitive periods are favored only early in ontogeny. These results are robust across different paradigms suggesting that empirical findings might be comparable despite different experimental designs.

Why Do Individuals Seek Information? A Selectionist Perspective

Several authors have proposed that mechanisms of adaptive behavior, and reinforcement learning in particular, can be explained by an innate tendency of individuals to seek information about the local environment. In this article, I argue that these approaches adhere to an essentialist view of learning that avoids the question why information seeking should be favorable in the first place. I propose a selectionist account of adaptive behavior that explains why individuals behave as if they had a tendency to seek information without resorting to essentialist explanations. I develop my argument using a formal selectionist framework for adaptive behavior, the multilevel model of behavioral selection (MLBS). The MLBS has been introduced recently as a formal theory of behavioral selection that links reinforcement learning to natural selection within a single unified model. I show that the MLBS implies an average gain in information about the availability of reinforcement. Formally, this means that behavior reaches an equilibrium state, if and only if the Fisher information of the conditional probability of reinforcement is maximized. This coincides with a reduction in the randomness of the expected environmental feedback as captured by the information theoretic concept of expected surprise (i.e., entropy). The main result is that behavioral selection maximizes the information about the expected fitness consequences of behavior, which, in turn, minimizes average surprise. In contrast to existing attempts to link adaptive behavior to information theoretic concepts (e.g., the free energy principle), neither information gain nor surprise minimization is treated as a first principle. Instead, the result is formally deduced from the MLBS and therefore constitutes a mathematical property of the more general principle of behavioral selection. Thus, if reinforcement learning is understood as a selection process, there is no need to assume an active agent with an innate tendency to seek information or minimize surprise. Instead, information gain and surprise minimization emerge naturally because it lies in the very nature of selection to produce order from randomness.

Reinforcement learning approaches to hippocampus-dependent flexible spatial navigation

Humans and non-human animals show great flexibility in spatial navigation, including the ability to return to specific locations based on as few as one single experience. To study spatial navigation in the laboratory, watermaze tasks, in which rats have to find a hidden platform in a pool of cloudy water surrounded by spatial cues, have long been used. Analogous tasks have been developed for human participants using virtual environments. Spatial learning in the watermaze is facilitated by the hippocampus. In particular, rapid, one-trial, allocentric place learning, as measured in the delayed-matching-to-place variant of the watermaze task, which requires rodents to learn repeatedly new locations in a familiar environment, is hippocampal dependent. In this article, we review some computational principles, embedded within a reinforcement learning framework, that utilise hippocampal spatial representations for navigation in watermaze tasks. We consider which key elements underlie their efficacy, and discuss their limitations in accounting for hippocampus-dependent navigation, both in terms of behavioural performance (i.e. how well do they reproduce behavioural measures of rapid place learning) and neurobiological realism (i.e. how well do they map to neurobiological substrates involved in rapid place learning). We discuss how an actor-critic architecture, enabling simultaneous assessment of the value of the current location and of the optimal direction to follow, can reproduce one-trial place learning performance as shown on watermaze and virtual delayed-matching-to-place tasks by rats and humans, respectively, if complemented with map-like place representations. The contribution of actor-critic mechanisms to delayed-matching-to-place performance is consistent with neurobiological findings implicating the striatum and hippocampo-striatal interaction in delayed-matching-to-place performance, given that the striatum has been associated with actor-critic mechanisms. Moreover, we illustrate that hierarchical computations embedded within an actor-critic architecture may help to account for aspects of flexible spatial navigation. The hierarchical reinforcement learning approach separates trajectory control via a temporal-difference error from goal selection via a goal prediction error and may account for flexible, trial-specific, navigation to familiar goal locations, as required in some arm-maze place memory tasks, although it does not capture one-trial learning of new goal locations, as observed in open field, including watermaze and virtual, delayed-matching-to-place tasks. Future models of one-shot learning of new goal locations, as observed on delayed-matching-to-place tasks, should incorporate hippocampal plasticity mechanisms that integrate new goal information with allocentric place representation, as such mechanisms are supported by substantial empirical evidence.

Who Reports Low Interactive Psychology Status? An Investigation Based on Chinese Coal Miners

In mine safety and health research, psychological issues have always been neglected. This paper aims to identify the psychological perceptions of workers with respect to the mine environment and interpersonal environment across the whole production system. A survey was designed that measured the miners' demographic details and perceptions of two affect-based interactions; three resource-based interactions for the manager, supervisor, co-worker; and three actual environment interactions. A total of 642 frontline coal miners from six mines located in six provinces in China completed the survey. The main results indicated that that miners reported low psychology status, especially those over 51 years old, with a monthly income of 2000-4000 and junior school education. Second, there was a high proportion of inferior value in environmental interactions. Meanwhile, the miners' interactions with their co-workers were perceived as the most positive and those with their managers as the least in interpersonal interactions. Third, there were significant differences in sub-dimension interactions (actual environment, resource-based, affect-based interactions) that certainly existed in these interactive roles. Additionally, the dissociated type of miners with manager and supervisor (low resource and affect-based interaction) reached 23.99~24.45%. This study revealed the inner psychological risk factors for safety and health work in coal mines and provides an essential guideline for mining industries.

A case for environmental statistics of early-life effects

There is enduring debate over the question of which early-life effects are adaptive and which ones are not. Mathematical modelling shows that early-life effects can be adaptive in environments that have particular statistical properties, such as reliable cues to current conditions and high autocorrelation of environmental states. However, few empirical studies have measured these properties, leading to an impasse. Progress, therefore, depends on research that quantifies cue reliability and autocorrelation of environmental parameters in real environments. These statistics may be different for social and non-social aspects of the environment. In this paper, we summarize evolutionary models of early-life effects. Then, we discuss empirical data on environmental statistics from a range of disciplines. We highlight cases where data on environmental statistics have been used to test competing explanations of early-life effects. We conclude by providing guidelines for new data collection and reflections on future directions.

This article is part of the theme issue ‘Developing differences: early-life effects and evolutionary medicine'.

Modeling the evolution of sensitive periods

In the past decade, there has been monumental progress in our understanding of the neurobiological basis of sensitive periods. Little is known, however, about the evolution of sensitive periods. Recent studies have started to address this gap. Biologists have built mathematical models exploring the environmental conditions in which sensitive periods are likely to evolve. These models investigate how mechanisms of plasticity can respond optimally to experience during an individual's lifetime. This paper discusses the central tenets, insights, and predictions of these models, in relation to empirical work on humans and other animals. We also discuss which future models are needed to improve the bridge between theory and data, advancing their synergy. Our paper is written in an accessible manner and for a broad audience. We hope our work will contribute to recently emerging connections between the fields of developmental neuroscience and evolutionary biology.

Cognitive gadgets: A provocative but flawed manifesto

The argument against innatism at the heart of Cognitive Gadgets is provocative but premature, and is vitiated by dichotomous thinking, interpretive double standards, and evidence cherry-picking. I illustrate my criticism by addressing the heritability of imitation and mindreading, the relevance of twin studies, and the meaning of cross-cultural differences in theory of mind development. Reaching an integrative understanding of genetic inheritance, plasticity, and learning is a formidable task that demands a more nuanced evolutionary approach.

Deep-Reinforcement Learning-Based Co-Evolution in a Predator-Prey System

Understanding or estimating the co-evolution processes is critical in ecology, but very challenging. Traditional methods are difficult to deal with the complex processes of evolution and to predict their consequences on nature. In this paper, we use the deep-reinforcement learning algorithms to endow the organism with learning ability, and simulate their evolution process by using the Monte Carlo simulation algorithm in a large-scale ecosystem. The combination of the two algorithms allows organisms to use experiences to determine their behavior through interaction with that environment, and to pass on experience to their offspring. Our research showed that the predators' reinforcement learning ability contributed to the stability of the ecosystem and helped predators obtain a more reasonable behavior pattern of coexistence with its prey. The reinforcement learning effect of prey on its own population was not as good as that of predators and increased the risk of extinction of predators. The inconsistent learning periods and speed of prey and predators aggravated that risk. The co-evolution of the two species had resulted in fewer numbers of their populations due to their potentially antagonistic evolutionary networks. If the learnable predators and prey invade an ecosystem at the same time, prey had an advantage. Thus, the proposed model illustrates the influence of learning mechanism on a predator-prey ecosystem and demonstrates the feasibility of predicting the behavior evolution in a predator-prey ecosystem using AI approaches.

Revisiting foraging approaches in neuroscience

Many complex real-world decisions, such as deciding which house to buy or whether to switch jobs, involve trying to maximize reward across a sequence of choices. Optimal Foraging Theory is well suited to study these kinds of choices because it provides formal models for reward-maximization in sequential situations. In this article, we review recent insights from foraging neuroscience, behavioral ecology, and computational modelling. We find that a commonly used approach in foraging neuroscience, in which choice items are encountered at random, does not reflect the way animals direct their foraging efforts in certain real-world settings, nor does it reflect efficient reward-maximizing behavior. Based on this, we propose that task designs allowing subjects to encounter choice items strategically will further improve the ecological validity of foraging approaches used in neuroscience, as well as give rise to new behavioral and neural predictions that deepen our understanding of sequential, value-based choice.

A computational perspective on social attachment

Humans depend on social relationships for survival and wellbeing throughout life. Yet, individuals differ markedly in their ability to form and maintain healthy social relationships. Here we use a simple mathematical model to formalize the contention that a person's attachment style is determined by what they learn from relationships early in life. For the sake of argument, we therefore discount individual differences in the innate personality or attachment style of a child, assuming instead that all children are simply born with an equivalent, generic, hardwired desire and instinct for social proximity, and a capacity to learn. In line with the evidence, this innate endowment incorporates both simple bonding instincts and a capacity for cognitively sophisticated beliefs and generalizations. Under this assumption, we then explore how distinct attachment styles might emerge through interaction with the child's early caregivers. Our central question is, how an apparently adaptive capacity to learn can yield enduring maladaptive attachment styles that generalize to new relationships. We believe extensions of our model will ultimately help clarify the complex interacting mechanisms - both acquired and innate - that underpin individual differences in attachment styles. While our model is relatively abstract, we also attempt some connection to known biological mechanisms of attachment.

Echoes of Early Life: Recent Insights From Mathematical Modeling

In the last decades, developmental origins of health and disease (DOHaD) has emerged as a central framework for studying early‐life effects, that is, the impact of fetal and early postnatal experience on adult functioning. Apace with empirical progress, theoreticians have built mathematical models that provide novel insights for DOHaD. This article focuses on three of these insights, which show the power of environmental noise (i.e., imperfect indicators of current and future conditions) in shaping development. Such noise can produce: (a) detrimental outcomes even in ontogenetically stable environments, (b) individual differences in sensitive periods, and (c) early‐life effects tailored to predicted future somatic states. We argue that these insights extend DOHaD and offer new research directions.