Reinforcement learning with artificial microswimmers

Light-Driven, Dynamic Assembly of Micron-To-Centimeter Parts, Micromachines and Microbot Swarms

This work describes light-driven assembly of dynamic formations and functional particle swarms controlled by appropriately programmed light patterns. The system capitalizes on the use of a fluidic bed whose low thermal conductivity assures that light-generated heat remains "localized" and sets strong convective flows in the immediate vicinity of the particles being irradiated. In this way, even low-power laser light or light from a desktop slide projector can be used to organize dynamic formations of objects spanning four orders of magnitude in size (from microns to centimeters) and over nine orders of magnitude in terms of mass. These dynamic assemblies include open-lattice structures with individual particles performing intricate translational and/or rotational motions, density-gradient particle arrays, nested architectures of mechanical components (e.g., planetary gears), or swarms of light-actuated microbots controlling assembly of other objects.

Swarm Autonomy: From Agent Functionalization to Machine Intelligence

Swarm behaviors are common in nature, where individual organisms collaborate via perception, communication, and adaptation. Emulating these dynamics, large groups of active agents can self-organize through localized interactions, giving rise to complex swarm behaviors, which exhibit potential for applications across various domains. This review presents a comprehensive summary and perspective of synthetic swarms, to bridge the gap between the microscale individual agents and potential applications of synthetic swarms. It is begun by examining active agents, the fundamental units of synthetic swarms, to understand the origins of their motility and functionality in the presence of external stimuli. Then inter-agent communications and agent-environment communications that contribute to the swarm generation are summarized. Furthermore, the swarm behaviors reported to date and the emergence of machine intelligence within these behaviors are reviewed. Eventually, the applications enabled by distinct synthetic swarms are summarized. By discussing the emergent machine intelligence in swarm behaviors, insights are offered into the design and deployment of autonomous synthetic swarms for real-world applications.

Learning how to find targets in the micro-world: the case of intermittent active Brownian particles

Finding the best strategy to minimize the time needed to find a given target is a crucial task both in nature and in reaching decisive technological advances. By considering learning agents able to switch their dynamics between standard and active Brownian motion, here we focus on developing effective target-search behavioral policies for microswimmers navigating a homogeneous environment and searching for targets of unknown position. We exploit projective simulation, a reinforcement learning algorithm, to acquire an efficient stochastic policy represented by the probability of switching the phase, i.e. the navigation mode, in response to the type and the duration of the current phase. Our findings reveal that the target-search efficiency increases with the particle's self-propulsion during the active phase and that, while the optimal duration of the passive case decreases monotonically with the activity, the optimal duration of the active phase displays a non-monotonic behavior.

AI-enhanced biomedical micro/nanorobots in microfluidics

Human beings encompass sophisticated microcirculation and microenvironments, incorporating a broad spectrum of microfluidic systems that adopt fundamental roles in orchestrating physiological mechanisms. In vitro recapitulation of human microenvironments based on lab-on-a-chip technology represents a critical paradigm to better understand the intricate mechanisms. Moreover, the advent of micro/nanorobotics provides brand new perspectives and dynamic tools for elucidating the complex process in microfluidics. Currently, artificial intelligence (AI) has endowed micro/nanorobots (MNRs) with unprecedented benefits, such as material synthesis, optimal design, fabrication, and swarm behavior. Using advanced AI algorithms, the motion control, environment perception, and swarm intelligence of MNRs in microfluidics are significantly enhanced. This emerging interdisciplinary research trend holds great potential to propel biomedical research to the forefront and make valuable contributions to human health. Herein, we initially introduce the AI algorithms integral to the development of MNRs. We briefly revisit the components, designs, and fabrication techniques adopted by robots in microfluidics with an emphasis on the application of AI. Then, we review the latest research pertinent to AI-enhanced MNRs, focusing on their motion control, sensing abilities, and intricate collective behavior in microfluidics. Furthermore, we spotlight biomedical domains that are already witnessing or will undergo game-changing evolution based on AI-enhanced MNRs. Finally, we identify the current challenges that hinder the practical use of the pioneering interdisciplinary technology.

Harnessing synthetic active particles for physical reservoir computing

The processing of information is an indispensable property of living systems realized by networks of active processes with enormous complexity. They have inspired many variants of modern machine learning, one of them being reservoir computing, in which stimulating a network of nodes with fading memory enables computations and complex predictions. Reservoirs are implemented on computer hardware, but also on unconventional physical substrates such as mechanical oscillators, spins, or bacteria often summarized as physical reservoir computing. Here we demonstrate physical reservoir computing with a synthetic active microparticle system that self-organizes from an active and passive component into inherently noisy nonlinear dynamical units. The self-organization and dynamical response of the unit are the results of a delayed propulsion of the microswimmer to a passive target. A reservoir of such units with a self-coupling via the delayed response can perform predictive tasks despite the strong noise resulting from the Brownian motion of the microswimmers. To achieve efficient noise suppression, we introduce a special architecture that uses historical reservoir states for output. Our results pave the way for the study of information processing in synthetic self-organized active particle systems.

Complex motion of steerable vesicular robots filled with active colloidal rods

While the collective motion of active particles has been studied extensively, effective strategies to navigate particle swarms without external guidance remain elusive. We introduce a method to control the trajectories of two-dimensional swarms of active rod-like particles by confining the particles to rigid bounding membranes (vesicles) with non-uniform curvature. We show that the propelling agents spontaneously form clusters at the membrane wall and collectively propel the vesicle, turning it into an active superstructure. To further guide the motion of the superstructure, we add discontinuous features to the rigid membrane boundary in the form of a kinked tip, which acts as a steering component to direct the motion of the vesicle. We report that the system's geometrical and material properties, such as the aspect ratio and Péclet number of the active rods as well as the kink angle and flexibility of the membrane, determine the stacking of active particles close to the kinked confinement and induce a diverse set of dynamical behaviors of the superstructure, including linear and circular motion both in the direction of, and opposite to, the kink. From a systematic study of these various behaviors, we design vesicles with switchable and reversible locomotions by tuning the confinement parameters. The observed phenomena suggest a promising mechanism for particle transportation and could be used as a basic element to navigate active matter through complex and tortuous environments.

Learning-based intelligent trajectory planning for auto navigation of magnetic robots

Introduction: Electromagnetically controlled small-scale robots show great potential in precise diagnosis, targeted delivery, and minimally invasive surgery. The automatic navigation of such robots could reduce human intervention, as well as the risk and difficulty of surgery. However, it is challenging to build a precise kinematics model for automatic robotic control because the controlling process is affected by various delays and complex environments. Method: Here, we propose a learning-based intelligent trajectory planning strategy for automatic navigation of magnetic robots without kinematics modeling. The Long Short-Term Memory (LSTM) neural network is employed to establish a global mapping relationship between the current sequence in the electromagnetic actuation system and the trajectory coordinates. Result: We manually control the robot to move on a curved path 50 times to form the training database to train the LSTM network. The trained LSTM network is validated to output the current sequence for automatically controlling the magnetic robot to move on the same curved path and the tortuous and branched new paths in simulated vascular tracks. Discussion: The proposed trajectory planning strategy is expected to impact the clinical applications of robots.

Realizing a deep reinforcement learning agent for real-time quantum feedback

Realizing the full potential of quantum technologies requires precise real-time control on time scales much shorter than the coherence time. Model-free reinforcement learning promises to discover efficient feedback strategies from scratch without relying on a description of the quantum system. However, developing and training a reinforcement learning agent able to operate in real-time using feedback has been an open challenge. Here, we have implemented such an agent for a single qubit as a sub-microsecond-latency neural network on a field-programmable gate array (FPGA). We demonstrate its use to efficiently initialize a superconducting qubit and train the agent based solely on measurements. Our work is a first step towards adoption of reinforcement learning for the control of quantum devices and more generally any physical device requiring low-latency feedback.

Collective foraging of active particles trained by reinforcement learning

Collective self-organization of animal groups is a recurring phenomenon in nature which has attracted a lot of attention in natural and social sciences. To understand how collective motion can be achieved without the presence of an external control, social interactions have been considered which regulate the motion and orientation of neighbors relative to each other. Here, we want to understand the motivation and possible reasons behind the emergence of such interaction rules using an experimental model system of light-responsive active colloidal particles (APs). Via reinforcement learning (RL), the motion of particles is optimized regarding their foraging behavior in presence of randomly appearing food sources. Although RL maximizes the rewards of single APs, we observe the emergence of collective behaviors within the particle group. The advantage of such collective strategy in context of foraging is to compensate lack of local information which strongly increases the robustness of the resulting policy. Our results demonstrate that collective behavior may not only result on the optimization of behaviors on the group level but may also arise from maximizing the benefit of individuals. Apart from a better understanding of collective behaviors in natural systems, these results may also be useful in context of the design of autonomous robotic systems.

Chemotaxis of an elastic flagellated microrobot

Machine learning algorithms offer a tool to boost mobility and flexibility of a synthetic microswimmer, hence may help us design truly smart microrobots. In this work, we design a two-gait microrobot swimming in circular or helical trajectory. It utilizes the coupling between flagellum elasticity and resistive force to change the characteristics of swimming trajectory. Leveraging a deep reinforcement learning (DRL) approach, we show that the microrobot can self-learn chemotactic motion autonomously (without heuristics) using only several current and historical chemoattractant concentration and curvature information. The learned strategy is more efficient than a human-devised shortsighted strategy and can be further greatly improved in a stochastic environment. Furthermore, in the helical trajectory case, if additional heuristic information of direction is supplemented to evaluate the strategy during the learning process, then a highly efficient strategy can be discovered by the DRL. The microrobot can quickly align the helix vector to the gradient direction using just several smart sequential gait switchings. The success for the efficient strategies depends on how much historical information is provided and also the steering angle step size of the microrobot. Our results provide useful guidance for the design and smart maneuver of synthetic spermlike microswimmers.

Optimal navigation of a smart active particle: directional and distance sensing

We employ Q learning, a variant of reinforcement learning, so that an active particle learns by itself to navigate on the fastest path toward a target while experiencing external forces and flow fields. As state variables, we use the distance and direction toward the target, and as action variables the active particle can choose a new orientation along which it moves with constant velocity. We explicitly investigate optimal navigation in a potential barrier/well and a uniform/ Poiseuille/swirling flow field. We show that Q learning is able to identify the fastest path and discuss the results. We also demonstrate that Q learning and applying the learned policy works when the particle orientation experiences thermal noise. However, the successful outcome strongly depends on the specific problem and the strength of noise.

Steering undulatory micro-swimmers in a fluid flow through reinforcement learning

This work aims at finding optimal navigation policies for thin, deformable microswimmers that progress in a viscous fluid by propagating a sinusoidal undulation along their slender body. These active filaments are embedded in a prescribed, non-homogeneous flow, in which their swimming undulations have to compete with the drifts, strains, and deformations inflicted by the outer velocity field. Such an intricate situation, where swimming and navigation are tightly bonded, is addressed using various methods of reinforcement learning. Each swimmer has only access to restricted information on its configuration and has to select accordingly an action among a limited set. The optimisation problem then consists in finding the policy leading to the most efficient displacement in a given direction. It is found that usual methods do not converge and this pitfall is interpreted as a combined consequence of the non-Markovianity of the decision process, together with the highly chaotic nature of the dynamics, which is responsible for high variability in learning efficiencies. Still, we provide an alternative method to construct efficient policies, which is based on running several independent realisations of Q-learning. This allows the construction of a set of admissible policies whose properties can be studied in detail and compared to assess their efficiency and robustness.

Learning to cooperate for low-Reynolds-number swimming: a model problem for gait coordination

Biological microswimmers can coordinate their motions to exploit their fluid environment-and each other-to achieve global advantages in their locomotory performance. These cooperative locomotion require delicate adjustments of both individual swimming gaits and spatial arrangements of the swimmers. Here we probe the emergence of such cooperative behaviors among artificial microswimmers endowed with artificial intelligence. We present the first use of a deep reinforcement learning approach to empower the cooperative locomotion of a pair of reconfigurable microswimmers. The AI-advised cooperative policy comprises two stages: an approach stage where the swimmers get in close proximity to fully exploit hydrodynamic interactions, followed a synchronization stage where the swimmers synchronize their locomotory gaits to maximize their overall net propulsion. The synchronized motions allow the swimmer pair to move together coherently with an enhanced locomotion performance unattainable by a single swimmer alone. Our work constitutes a first step toward uncovering intriguing cooperative behaviors of smart artificial microswimmers, demonstrating the vast potential of reinforcement learning towards intelligent autonomous manipulations of multiple microswimmers for their future biomedical and environmental applications.

Learning to swim efficiently in a nonuniform flow field

Microswimmers can acquire information on the surrounding fluid by sensing mechanical queues. They can then navigate in response to these signals. We analyze this navigation by combining deep reinforcement learning with direct numerical simulations to resolve the hydrodynamics. We study how local and nonlocal information can be used to train a swimmer to achieve particular swimming tasks in a nonuniform flow field, in particular, a zigzag shear flow. The swimming tasks are (1) learning how to swim in the vorticity direction, (2) learning how to swim in the shear-gradient direction, and (3) learning how to swim in the shear-flow direction. We find that access to laboratory frame information on the swimmer's instantaneous orientation is all that is required in order to reach the optimal policy for tasks (1) and (2). However, information on both the translational and rotational velocities seems to be required to accomplish task (3). Inspired by biological microorganisms, we also consider the case where the swimmers sense local information, i.e., surface hydrodynamic forces, together with a signal direction. This might correspond to gravity or, for microorganisms with light sensors, a light source. In this case, we show that the swimmer can reach a comparable level of performance to that of a swimmer with access to laboratory frame variables. We also analyze the role of different swimming modes, i.e., pusher, puller, and neutral.

Light, Matter, Action: Shining Light on Active Matter

Light carries energy and momentum. It can therefore alter the motion of objects on the atomic to astronomical scales. Being widely available, readily controllable, and broadly biocompatible, light is also an ideal tool to propel microscopic particles, drive them out of thermodynamic equilibrium, and make them active. Thus, light-driven particles have become a recent focus of research in the field of soft active matter. In this Perspective, we discuss recent advances in the control of soft active matter with light, which has mainly been achieved using light intensity. We also highlight some first attempts to utilize light's additional properties, such as its wavelength, polarization, and momentum. We then argue that fully exploiting light with all of its properties will play a critical role in increasing the level of control over the actuation of active matter as well as the flow of light itself through it. This enabling step will advance the design of soft active matter systems, their functionalities, and their transfer toward technological applications.

Biodegradable Microrobots and Their Biomedical Applications: A Review

During recent years, microrobots have drawn extensive attention owing to their good controllability and great potential in biomedicine. Powered by external physical fields or chemical reactions, these untethered microdevices are promising candidates for in vivo complex tasks, such as targeted delivery, imaging and sensing, tissue engineering, hyperthermia, and assisted fertilization, among others. However, in clinical use, the biodegradability of microrobots is significant for avoiding toxic residue in the human body. The selection of biodegradable materials and the corresponding in vivo environment needed for degradation are increasingly receiving attention in this regard. This review aims at analyzing different types of biodegradable microrobots by critically discussing their advantages and limitations. The chemical degradation mechanisms behind biodegradable microrobots and their typical applications are also thoroughly investigated. Furthermore, we examine their feasibility and deal with the in vivo suitability of different biodegradable microrobots in terms of their degradation mechanisms; pathological environments; and corresponding biomedical applications, especially targeted delivery. Ultimately, we highlight the prevailing obstacles and perspective solutions, ranging from their manufacturing methods, control of movement, and degradation rate to insufficient and limited in vivo tests, that could be of benefit to forthcoming clinical applications.

Accelerating the Design of Self-Guided Microrobots in Time-Varying Magnetic Fields

Mobile robots combine sensory information with mechanical actuation to move autonomously through structured environments and perform specific tasks. The miniaturization of such robots to the size of living cells is actively pursued for applications in biomedicine, materials science, and environmental sustainability. Existing microrobots based on field-driven particles rely on knowledge of the particle position and the target destination to control particle motion through fluid environments. Often, however, these external control strategies are challenged by limited information and global actuation where a common field directs multiple robots with unknown positions. In this Perspective, we discuss how time-varying magnetic fields can be used to encode the self-guided behaviors of magnetic particles conditioned on local environmental cues. Programming these behaviors is framed as a design problem: we seek to identify the design variables (e.g., particle shape, magnetization, elasticity, stimuli-response) that achieve the desired performance in a given environment. We discuss strategies for accelerating the design process using automated experiments, computational models, statistical inference, and machine learning approaches. Based on the current understanding of field-driven particle dynamics and existing capabilities for particle fabrication and actuation, we argue that self-guided microrobots with potentially transformative capabilities are close at hand.

Persistent motion of a Brownian particle subject to repulsive feedback with time delay

Based on analytical and numerical calculations we study the dynamics of an overdamped colloidal particle moving in two dimensions under time-delayed, nonlinear feedback control. Specifically, the particle is subject to a force derived from a repulsive Gaussian potential depending on the difference between its instantaneous position, r(t), and its earlier position r(t-τ), where τ is the delay time. Considering first the deterministic case, we provide analytical results for both the case of small displacements and the dynamics at long times. In particular, at appropriate values of the feedback parameters, the particle approaches a steady state with a constant, nonzero velocity whose direction is constant as well. In the presence of noise, the direction of motion becomes randomized at long times, but the (numerically obtained) velocity autocorrelation still reveals some persistence of motion. Moreover, the mean-squared displacement (MSD) reveals a mixed regime at intermediate times with contributions of both ballistic motion and diffusive translational motion, allowing us to extract an estimate for the effective propulsion velocity in presence of noise. We then analyze the data in terms of exact, known results for the MSD of active Brownian particles. The comparison indeed indicates a strong similarity between the dynamics of the particle under repulsive delayed feedback and active motion. This relation carries over to the behavior of the long-time diffusion coefficient D_{eff} which, similarly to active motion, is strongly enhanced compared to the free case. Finally, we show that, for small delays, D_{eff} can be estimated analytically.

Advanced medical micro-robotics for early diagnosis and therapeutic interventions

Recent technological advances in micro-robotics have demonstrated their immense potential for biomedical applications. Emerging micro-robots have versatile sensing systems, flexible locomotion and dexterous manipulation capabilities that can significantly contribute to the healthcare system. Despite the appreciated and tangible benefits of medical micro-robotics, many challenges still remain. Here, we review the major challenges, current trends and significant achievements for developing versatile and intelligent micro-robotics with a focus on applications in early diagnosis and therapeutic interventions. We also consider some recent emerging micro-robotic technologies that employ synthetic biology to support a new generation of living micro-robots. We expect to inspire future development of micro-robots toward clinical translation by identifying the roadblocks that need to be overcome.

Spontaneous vortex formation by microswimmers with retarded attractions

Collective states of inanimate particles self-assemble through physical interactions and thermal motion. Despite some phenomenological resemblance, including signatures of criticality, the autonomous dynamics that binds motile agents into flocks, herds, or swarms allows for much richer behavior. Low-dimensional models have hinted at the crucial role played in this respect by perceived information, decision-making, and feedback, implying that the corresponding interactions are inevitably retarded. Here we present experiments on spherical Brownian microswimmers with delayed self-propulsion toward a spatially fixed target. We observe a spontaneous symmetry breaking to a transiently chiral dynamical state and concomitant critical behavior that do not rely on many-particle cooperativity. By comparison with the stochastic delay differential equation of motion of a single swimmer, we pinpoint the delay-induced effective synchronization of the swimmers with their own past as the key mechanism. Increasing numbers of swimmers self-organize into layers with pro- and retrograde orbital motion, synchronized and stabilized by steric, phoretic, and hydrodynamic interactions. Our results demonstrate how even most simple retarded interactions can foster emergent complex adaptive behavior in small active-particle ensembles.

Controlled propulsion of micro/nanomotors: operational mechanisms, motion manipulation and potential biomedical applications

Inspired by natural mobile microorganisms, researchers have developed micro/nanomotors (MNMs) that can autonomously move by transducing different kinds of energies into kinetic energy. The rapid development of MNMs has created tremendous opportunities for biomedical fields including diagnostics, therapeutics, and theranostics. Although the great progress has been made in MNM research, at a fundamental level, the accepted propulsion mechanisms are still a controversial matter. In practical applications such as precision nanomedicine, the precise control of the motion, including the speed and directionality, of MNMs is also important, which makes advanced motion manipulation desirable. Very recently, diverse MNMs with different propulsion strategies, morphologies, sizes, porosities and chemical structures have been fabricated and applied for various uses. Herein, we thoroughly summarize the physical principles behind propulsion strategies, as well as the recent advances in motion manipulation methods and relevant biomedical applications of these MNMs. The current challenges in MNM research are also discussed. We hope this review can provide a bird's eye overview of the MNM research and inspire researchers to create novel and more powerful MNMs.

Active matter in space

In the last 20 years, active matter has been a highly dynamic field of research, bridging fundamental aspects of non-equilibrium thermodynamics with applications to biology, robotics, and nano-medicine. Active matter systems are composed of units that can harvest and harness energy and information from their environment to generate complex collective behaviours and forms of self-organisation. On Earth, gravity-driven phenomena (such as sedimentation and convection) often dominate or conceal the emergence of these dynamics, especially for soft active matter systems where typical interactions are of the order of the thermal energy. In this review, we explore the ongoing and future efforts to study active matter in space, where low-gravity and microgravity conditions can lift some of these limitations. We envision that these studies will help unify our understanding of active matter systems and, more generally, of far-from-equilibrium physics both on Earth and in space. Furthermore, they will also provide guidance on how to use, process and manufacture active materials for space exploration and colonisation.

Signal propagation and linear response in the delay Vicsek model

Retardation between sensation and action is an inherent biological trait. Here we study its effect in the Vicsek model, which is a paradigmatic swarm model. We find that (1) a discrete time delay in the orientational interactions diminishes the ability of strongly aligned swarms to follow a leader and, in return, increases their stability against random orientation fluctuations; (2) both longer delays and higher speeds favor ballistic over diffusive spreading of information (orientation) through the swarm; (3) for short delays, the mean change in the total orientation (the order parameter) scales linearly in a small orientational bias of the leaders and inversely in the delay time, while its variance first increases and then saturates with increasing delays; and (4) the linear response breaks down when orientation conservation is broken.

Smart Magnetic Microrobots Learn to Swim with Deep Reinforcement Learning

Swimming microrobots are increasingly developed with complex materials and dynamic shapes and are expected to operate in complex environments in which the system dynamics are difficult to model and positional control of the microrobot is not straightforward to achieve. Deep reinforcement learning is a promising method of autonomously developing robust controllers for creating smart microrobots, which can adapt their behavior to operate in uncharacterized environments without the need to model the system dynamics. This article reports the development of a smart helical magnetic hydrogel microrobot that uses the soft actor critic reinforcement learning algorithm to autonomously derive a control policy which allows the microrobot to swim through an uncharacterized biomimetic fluidic environment under control of a time varying magnetic field generated from a three-axis array of electromagnets. The reinforcement learning agent learned successful control policies from both state vector input and raw images, and the control policies learned by the agent recapitulated the behavior of rationally designed controllers based on physical models of helical swimming microrobots. Deep reinforcement learning applied to microrobot control is likely to significantly expand the capabilities of the next generation of microrobots.

Diffusion coefficient and power spectrum of active particles with a microscopically reversible mechanism of self-propelling

Catalytically active macromolecules are envisioned as key building blocks in development of artificial nanomotors. However, theory and experiments report conflicting findings regarding their dynamics. The lack of consensus is mostly caused by a limited understanding of specifics of self-propulsion mechanisms at the nanoscale. Here, we study a generic model of a self-propelled nanoparticle that does not rely on a particular mechanism. Instead, its main assumption is the fundamental symmetry of microscopic dynamics of chemical reactions: the principle of microscopic reversibility. Significant consequences of this assumption arise if we subject the particle to an action of an external time-periodic force. The particle diffusion coefficient is then enhanced compared to the unbiased dynamics. The enhancement can be controlled by the force amplitude and frequency. We also derive the power spectrum of particle trajectories. Among new effects stemming from the microscopic reversibility are the enhancement of the spectrum at all frequencies and sigmoid-shaped transitions and a peak at characteristic frequencies of rotational diffusion and external forcing. The microscopic reversibility is a generic property of a broad class of chemical reactions, therefore we expect that the presented results will motivate new experimental studies aimed at testing of our predictions. This could provide new insights into dynamics of catalytic macromolecules.

Heat-Mediated Optical Manipulation

Progress in optical manipulation has stimulated remarkable advances in a wide range of fields, including materials science, robotics, medical engineering, and nanotechnology. This Review focuses on an emerging class of optical manipulation techniques, termed heat-mediated optical manipulation. In comparison to conventional optical tweezers that rely on a tightly focused laser beam to trap objects, heat-mediated optical manipulation techniques exploit tailorable optothermo-matter interactions and rich mass transport dynamics to enable versatile control of matter of various compositions, shapes, and sizes. In addition to conventional tweezing, more distinct manipulation modes, including optothermal pulling, nudging, rotating, swimming, oscillating, and walking, have been demonstrated to enhance the functionalities using simple and low-power optics. We start with an introduction to basic physics involved in heat-mediated optical manipulation, highlighting major working mechanisms underpinning a variety of manipulation techniques. Next, we categorize the heat-mediated optical manipulation techniques based on different working mechanisms and discuss working modes, capabilities, and applications for each technique. We conclude this Review with our outlook on current challenges and future opportunities in this rapidly evolving field of heat-mediated optical manipulation.

Reinforcement learning reveals fundamental limits on the mixing of active particles

The control of far-from-equilibrium physical systems, including active materials, requires advanced control strategies due to the non-linear dynamics and long-range interactions between particles, preventing explicit solutions to optimal control problems. In such situations, Reinforcement Learning (RL) has emerged as an approach to derive suitable control strategies. However, for active matter systems, it is an important open question how the mathematical structure and the physical properties determine the tractability of RL. In this paper, we demonstrate that RL can only find good mixing strategies for active matter systems that combine attractive and repulsive interactions. Using analytic results from dynamical systems theory, we show that combining both interaction types is indeed necessary for the existence of mixing-inducing hyperbolic dynamics and therefore the ability of RL to find homogeneous mixing strategies. In particular, we show that for drag-dominated translational-invariant particle systems, mixing relies on combined attractive and repulsive interactions. Therefore, our work demonstrates which experimental developments need to be made to make protein-based active matter applicable, and it provides some classification of microscopic interactions based on macroscopic behavior.

Hunting active Brownian particles: Learning optimal behavior

We numerically study active Brownian particles that can respond to environmental cues through a small set of actions (switching their motility and turning left or right with respect to some direction) which are motivated by recent experiments with colloidal self-propelled Janus particles. We employ reinforcement learning to find optimal mappings between the state of particles and these actions. Specifically, we first consider a predator-prey situation in which prey particles try to avoid a predator. Using as reward the squared distance from the predator, we discuss the merits of three state-action sets and show that turning away from the predator is the most successful strategy. We then remove the predator and employ as collective reward the local concentration of signaling molecules exuded by all particles and show that aligning with the concentration gradient leads to chemotactic collapse into a single cluster. Our results illustrate a promising route to obtain local interaction rules and design collective states in active matter.

Stabilizing viscous extensional flows using reinforcement learning

The four-roll mill, wherein four identical cylinders undergo rotation of identical magnitude but alternate signs, was originally proposed by G. I. Taylor to create local extensional flows and study their ability to deform small liquid drops. Since an extensional flow has an unstable eigendirection, a drop located at the flow stagnation point will have a tendency to escape. This unstable dynamics can, however, be stabilized using, e.g., a modulation of the rotation rates of the cylinders. Here we use reinforcement learning, a branch of machine learning devoted to the optimal selection of actions based on cumulative rewards, in order to devise a stabilization algorithm for the four-roll mill flow. The flow is modelled as the linear superposition of four two-dimensional rotlets and the drop is treated as a rigid spherical particle smaller than all other length scales in the problem. Unlike previous attempts to devise control, we take a probabilistic approach whereby speed adjustments are drawn from a probability density function whose shape is improved over time via a form of gradient ascent know as actor-critic method. With enough training, our algorithm is able to precisely control the drop and keep it close to the stagnation point for as long as needed. We explore the impact of the physical and learning parameters on the effectiveness of the control and demonstrate the robustness of the algorithm against thermal noise. We finally show that reinforcement learning can provide a control algorithm effective for all initial positions and that can be adapted to limit the magnitude of the flow extension near the position of the drop.

Finite-Size Scaling at the Edge of Disorder in a Time-Delay Vicsek Model

Living many-body systems often exhibit scale-free collective behavior reminiscent of thermal critical phenomena. But their mutual interactions are inevitably retarded due to information processing and delayed actuation. We numerically investigate the consequences for the finite-size scaling in the Vicsek model of motile active matter. A growing delay time initially facilitates but ultimately impedes collective ordering and turns the dynamical scaling from diffusive to ballistic. It provides an alternative explanation of swarm traits previously attributed to inertia.

Control and controllability of microswimmers by a shearing flow

With the continuing rapid development of artificial microrobots and active particles, questions of microswimmer guidance and control are becoming ever more relevant and prevalent. In both the applications and theoretical study of such microscale swimmers, control is often mediated by an engineered property of the swimmer, such as in the case of magnetically propelled microrobots. In this work, we will consider a modality of control that is applicable in more generality, effecting guidance via modulation of a background fluid flow. Here, considering a model swimmer in a commonplace flow and simple geometry, we analyse and subsequently establish the efficacy of flow-mediated microswimmer positional control, later touching upon a question of optimal control. Moving beyond idealized notions of controllability and towards considerations of practical utility, we then evaluate the robustness of this control modality to sources of variation that may be present in applications, examining in particular the effects of measurement inaccuracy and rotational noise. This exploration gives rise to a number of cautionary observations, which, overall, demonstrate the need for the careful assessment of both policy and behavioural robustness when designing control schemes for use in practice.

Microswimmers learning chemotaxis with genetic algorithms

Various microorganisms and some mammalian cells are able to swim in viscous fluids by performing nonreciprocal body deformations, such as rotating attached flagella or by distorting their entire body. In order to perform chemotaxis (i.e., to move toward and to stay at high concentrations of nutrients), they adapt their swimming gaits in a nontrivial manner. Here, we propose a computational model, which features autonomous shape adaptation of microswimmers moving in one dimension toward high field concentrations. As an internal decision-making machinery, we use artificial neural networks, which control the motion of the microswimmer. We present two methods to measure chemical gradients, spatial and temporal sensing, as known for swimming mammalian cells and bacteria, respectively. Using the genetic algorithm NeuroEvolution of Augmenting Topologies, surprisingly simple neural networks evolve. These networks control the shape deformations of the microswimmers and allow them to navigate in static and complex time-dependent chemical environments. By introducing noisy signal transmission in the neural network, the well-known biased run-and-tumble motion emerges. Our work demonstrates that the evolution of a simple and interpretable internal decision-making machinery coupled to the environment allows navigation in diverse chemical landscapes. These findings are of relevance for intracellular biochemical sensing mechanisms of single cells or for the simple nervous system of small multicellular organisms such as Caenorhabditis elegans.

Artificial microswimmers get smart

Reinforcement learning enables microswimmers to navigate through noisy and unexplored real-world environments.