Hunting active Brownian particles: Learning optimal behavior

Smart active particles learn and transcend bacterial foraging strategies

Throughout evolution, bacteria and other microorganisms have learned efficient foraging strategies that exploit characteristic properties of their unknown environment. While much research has been devoted to the exploration of statistical models describing the dynamics of foraging bacteria and other (micro-) organisms, little is known, regarding the question of how good the learned strategies actually are. This knowledge gap is largely caused by the absence of methods allowing to systematically develop alternative foraging strategies to compare with. In the present work, we use deep reinforcement learning to show that a smart run-and-tumble agent, which strives to find nutrients for its survival, learns motion patterns that are remarkably similar to the trajectories of chemotactic bacteria. Strikingly, despite this similarity, we also find interesting differences between the learned tumble rate distribution and the one that is commonly assumed for the run and tumble model. We find that these differences equip the agent with significant advantages regarding its foraging and survival capabilities. Our results uncover a generic route to use deep reinforcement learning for discovering search and collection strategies that exploit characteristic but initially unknown features of the environment. These results can be used, e.g., to program future microswimmers, nanorobots, and smart active particles for tasks like searching for cancer cells, micro-waste collection, or environmental remediation.

Optimal navigation of a smart active particle: directional and distance sensing

We employ Q learning, a variant of reinforcement learning, so that an active particle learns by itself to navigate on the fastest path toward a target while experiencing external forces and flow fields. As state variables, we use the distance and direction toward the target, and as action variables the active particle can choose a new orientation along which it moves with constant velocity. We explicitly investigate optimal navigation in a potential barrier/well and a uniform/ Poiseuille/swirling flow field. We show that Q learning is able to identify the fastest path and discuss the results. We also demonstrate that Q learning and applying the learned policy works when the particle orientation experiences thermal noise. However, the successful outcome strongly depends on the specific problem and the strength of noise.

Ordering through learning in two-dimensional Ising spins

We study two-dimensional Ising spins, evolving through reinforcement learning using their state, action, and reward. The state of a spin is defined by whether it is in the majority or minority with its nearest neighbors. The spin updates its state using an ε-greedy algorithm. The parameter ε plays a role equivalent to the temperature in the Ising model. We find a phase transition from long-ranged ordered to a disordered state as we tune ε from small to large values. In analogy with the phase transition in the Ising model, we calculate the critical ε and the three critical exponents β, γ, ν of magnetization, susceptibility, and correlation length, respectively. A hyperscaling relation dν=2β+γ is obtained between the three exponents. The system is studied for different learning rates. The exponents approach the exact values for a two-dimensional Ising model for lower learning rates.