Solution of classical evolutionary models in the limit when the diffusion approximation breaks down

Quantifying the stochasticity of policy parameters in reinforcement learning problems

The stochastic dynamics of reinforcement learning is studied using a master equation formalism. We consider two different problems-Q learning for a two-agent game and the multiarmed bandit problem with policy gradient as the learning method. The master equation is constructed by introducing a probability distribution over continuous policy parameters or over both continuous policy parameters and discrete state variables (a more advanced case). We use a version of the moment closure approximation to solve for the stochastic dynamics of the models. Our method gives accurate estimates for the mean and the (co)variance of policy variables. For the case of the two-agent game, we find that the variance terms are finite at steady state and derive a system of algebraic equations for computing them directly.

Allele fixation probability in a Moran model with fluctuating fitness landscapes

Evolution on changing fitness landscapes (seascapes) is an important problem in evolutionary biology. We consider the Moran model of finite population evolution with selection in a randomly changing, dynamic environment. In the model, each individual has one of the two alleles, wild type or mutant. We calculate the fixation probability by making a proper ansatz for the logarithm of fixation probabilities. This method has been used previously to solve the analogous problem for the Wright-Fisher model. The fixation probability is related to the solution of a third-order algebraic equation (for the logarithm of fixation probability). We consider the strong interference of landscape fluctuations, sampling, and selection when the fixation process cannot be described by the mean fitness. Such an effect appears if the mutant allele has a higher fitness in one landscape and a lower fitness in another, compared with the wild type, and the product of effective population size and fitness is large. We provide a generalization of the Kimura formula for the fixation probability that applies to these cases. When the mutant allele has a fitness (dis-)advantage in both landscapes, the fixation probability is described by the mean fitness.

Looking for the optimal rate of recombination for evolutionary dynamics

We consider many-site mutation-recombination models of evolution with selection. We are looking for situations where the recombination increases the mean fitness of the population, and there is an optimal recombination rate. We found two fitness landscapes supporting such nonmonotonic behavior of the mean fitness versus the recombination rate. The first case is related to the evolution near the error threshold on a neutral-network-like fitness landscape, for moderate genome lengths and large population. The more realistic case is the second one, in which we consider the evolutionary dynamics of a finite population on a rugged fitness landscape (the smooth fitness landscape plus some random contributions to the fitness). We also give the solution to the horizontal gene transfer model in the case of asymmetric mutations. To obtain nonmonotonic behavior for both mutation and recombination, we need a specially designed (ideal) fitness landscape.

Crossing fitness canyons by a finite population

We consider the Wright-Fisher model of the finite population evolution on a fitness landscape defined in the sequence space by a path of nearly neutral mutations. We study a specific structure of the fitness landscape: One of the intermediate mutations on the mutation path results in either a large fitness value (climbing up a fitness hill) or a low fitness value (crossing a fitness canyon), the rest of the mutations besides the last one are neutral, and the last sequence has much higher fitness than any intermediate sequence. We derive analytical formulas for the first arrival time of the mutant with two point mutations. For the first arrival problem for the further mutants in the case of canyon crossing, we analytically deduce how the mean first arrival time scales with the population size and fitness difference. The location of the canyon on the path of sequences has a crucial role. If the canyon is at the beginning of the path, then it significantly prolongs the first arrival time; otherwise it just slightly changes it. Furthermore, the fitness hill at the beginning of the path strongly prolongs the arrival time period; however, the hill located near the end of the path shortens it. We optimize the first arrival time by applying a nonzero selection to the intermediate sequences. We extend our results and provide a scaling for the valley crossing time via the depth of the canyon and population size in the case of a fitness canyon at the first position. Our approach is useful for understanding some complex evolution systems, e.g., the evolution of cancer.