An evolutionary model of reinforcer value

Model-based estimates for operant selection

We present a new methodology to partition different sources of behavior change within a selectionist framework based on the Price equation-the multilevel model of behavioral selection. The multilevel model of behavioral selection provides a theoretical background to describe behavior change in terms of operant selection. Operant selection is formally captured by the covariance-based law of effect and accounts for all changes in individual behavior that involve a covariance between behavior and predictors of evolutionary fitness (e.g., food). In this article, we show how the covariance-based law of effect may be applied to different components of operant behavior (e.g., allocation, speed, and accuracy of responding), thereby providing quantitative estimates for various selection effects affecting behavior change using data from a published learning experiment with pigeons.

Behavioral selection in structured populations

The multilevel model of behavioral selection (MLBS) by Borgstede and Eggert (Behav Process 186:104370. 10.1016/j.beproc.2021.104370 , 2021) provides a formal framework that integrates reinforcement learning with natural selection using an extended Price equation. However, the MLBS is so far only formulated for homogeneous populations, thereby excluding all sources of variation between individuals. This limitation is of primary theoretical concern because any application of the MLBS to real data requires to account for variation between individuals. In this paper, I extend the MLBS to account for inter-individual variation by dividing the population into homogeneous sub-populations and including class-specific reproductive values as weighting factors for an individual's evolutionary fitness. The resulting formalism closes the gap between the theoretical underpinnings of behavioral selection and the application of the theory to empirical data, which naturally includes inter-individual variation. Furthermore, the extended MLBS is used to establish an explicit connection between the dynamics of learning and the maximization of individual fitness. These results expand the scope of the MLBS as a general theoretical framework for the quantitative analysis of learning and evolution.

The Popcorn Illusion

A popcorn popping is almost magical. And yet, the science of popcorn is safe and clear about the steps until the pop: the components, processes, and results of making popcorn. Nature has its own way to produce surprise in the form of "pops" (i.e., emergence, qualitative shifts). Emergent features spread throughout the life of taxa and individuals. A pop can be sudden and chaotic. And so is creativity. There is no incompatibility between creativity and naturalistic endeavors in science. Creativity is no god given gift blown inside humans. When creativity is defined by originality and spontaneity, it describes a feature with no past or present. I briefly summarize how one can see non-random innovation, no free occurring spontaneity, and non-heuristic effectiveness as features of behaviors that are not necessarily considered creative. Those three features reveal how traditional views of creativity undermine its real determiners and how it can be objectively defined and observed.

Meaningful measurement requires substantive formal theory

In this article, we take the opportunity to elaborate on some aspects of our article “Squaring the Circle: From Latent Variables to Theory-Based Measurement” (Borgstede & Eggert, 2023) that gave rise to the concerns uttered by Hasselman (2023) and Slaney (2023), and to clarify why we think that theory-based measurement is indeed necessary and sufficient for the establishment of meaningful psychological measurement procedures. Moreover, we will illustrate how theory-based measurement might be accomplished in psychology by means of an example from behavioral selection theory.

Squaring the circle: From latent variables to theory-based measurement

Psychometrics builds on the fundamental premise that psychological attributes are unobservable and need to be inferred from observable behavior. Consequently, psychometric procedures consist primarily in applying latent variable modeling, which statistically relates latent variables to manifest variables. However, latent variable modeling falls short of providing a theoretically sound definition of psychological attributes. Whereas in a pragmatic interpretation of latent variable modeling latent variables cannot represent psychological attributes at all, a realist interpretation of latent variable modeling implies that latent variables are empty placeholders for unknown attributes. The authors argue that psychological attributes can only be identified if they are defined within the context of substantive formal theory. Building on the structuralist view of scientific theories, they show that any successful application of such a theory necessarily produces specific values for the theoretical terms that are defined within the theory. Therefore, substantive formal theory is both necessary and sufficient for psychological measurement.

Why Do Individuals Seek Information? A Selectionist Perspective

Several authors have proposed that mechanisms of adaptive behavior, and reinforcement learning in particular, can be explained by an innate tendency of individuals to seek information about the local environment. In this article, I argue that these approaches adhere to an essentialist view of learning that avoids the question why information seeking should be favorable in the first place. I propose a selectionist account of adaptive behavior that explains why individuals behave as if they had a tendency to seek information without resorting to essentialist explanations. I develop my argument using a formal selectionist framework for adaptive behavior, the multilevel model of behavioral selection (MLBS). The MLBS has been introduced recently as a formal theory of behavioral selection that links reinforcement learning to natural selection within a single unified model. I show that the MLBS implies an average gain in information about the availability of reinforcement. Formally, this means that behavior reaches an equilibrium state, if and only if the Fisher information of the conditional probability of reinforcement is maximized. This coincides with a reduction in the randomness of the expected environmental feedback as captured by the information theoretic concept of expected surprise (i.e., entropy). The main result is that behavioral selection maximizes the information about the expected fitness consequences of behavior, which, in turn, minimizes average surprise. In contrast to existing attempts to link adaptive behavior to information theoretic concepts (e.g., the free energy principle), neither information gain nor surprise minimization is treated as a first principle. Instead, the result is formally deduced from the MLBS and therefore constitutes a mathematical property of the more general principle of behavioral selection. Thus, if reinforcement learning is understood as a selection process, there is no need to assume an active agent with an innate tendency to seek information or minimize surprise. Instead, information gain and surprise minimization emerge naturally because it lies in the very nature of selection to produce order from randomness.

The formal foundation of an evolutionary theory of reinforcement

Reinforcement learning is often described by analogy to natural selection. However, there is no coherent theory relating reinforcement learning to evolution within a single formal model of selection. This paper provides the formal foundation of such a unified theory. The model is based on the most general description of natural selection as given by the Price equation. We extend the Price equation to cover reinforcement learning as the result of a behavioral selection process within individuals and relate it to the principle of natural selection via the concept of statistical fitness predictors by means of a multilevel model of behavioral selection. The main result is the covariance-based law of effect, which describes reinforcement learning on a molar level by means of the covariance between behavioral allocation and a statistical fitness predictor. We further demonstrate how this abstract principle can be applied to derive theoretical explanations of various empirical findings, like conditioned reinforcement, blocking, matching and response deprivation. Our model is the first to apply the abstract principle of selection to derive a unified description of reinforcement learning and natural selection within a single model. It provides a general analytical tool for behavioral psychology in a similar way that the theory of natural selection does for evolutionary biology. We thus lay the formal foundation of a general theory of reinforcement as the result of behavioral selection on multiple levels.