Control of neural systems at multiple scales using model-free, deep reinforcement learning

Data-driven control of oscillator networks with population-level measurement

Controlling complex networks of nonlinear limit-cycle oscillators is an important problem pertinent to various applications in engineering and natural sciences. While in recent years the control of oscillator populations with comprehensive biophysical models or simplified models, e.g., phase models, has seen notable advances, learning appropriate controls directly from data without prior model assumptions or pre-existing data remains a challenging and less developed area of research. In this paper, we address this problem by leveraging the network's current dynamics to iteratively learn an appropriate control online without constructing a global model of the system. We illustrate through a range of numerical simulations that the proposed technique can effectively regulate synchrony in various oscillator networks after a small number of trials using only one input and one noisy population-level output measurement. We provide a theoretical analysis of our approach, illustrate its robustness to system variations, and compare its performance with existing model-based and data-driven approaches.

Optimal synchronization in pulse-coupled oscillator networks using reinforcement learning

Spontaneous synchronization is ubiquitous in natural and man-made systems. It underlies emergent behaviors such as neuronal response modulation and is fundamental to the coordination of robot swarms and autonomous vehicle fleets. Due to its simplicity and physical interpretability, pulse-coupled oscillators has emerged as one of the standard models for synchronization. However, existing analytical results for this model assume ideal conditions, including homogeneous oscillator frequencies and negligible coupling delays, as well as strict requirements on the initial phase distribution and the network topology. Using reinforcement learning, we obtain an optimal pulse-interaction mechanism (encoded in phase response function) that optimizes the probability of synchronization even in the presence of nonideal conditions. For small oscillator heterogeneities and propagation delays, we propose a heuristic formula for highly effective phase response functions that can be applied to general networks and unrestricted initial phase distributions. This allows us to bypass the need to relearn the phase response function for every new network.

Data-Driven Control of Neuronal Networks with Population-Level Measurement

Controlling complex networks of nonlinear neurons is an important problem pertinent to various applications in engineering and natural sciences. While in recent years the control of neural populations with comprehensive biophysical models or simplified models, e.g., phase models, has seen notable advances, learning appropriate controls directly from data without any model assumptions remains a challenging and less developed area of research. In this paper, we address this problem by leveraging the network's local dynamics to iteratively learn an appropriate control without constructing a global model of the system. The proposed technique can effectively regulate synchrony in a neuronal network using only one input and one noisy population-level output measurement. We provide a theoretical analysis of our approach and illustrate its robustness to system variations and its generalizability to accommodate various physical constraints, such as charge-balanced inputs.

An in-silico framework for modeling optimal control of neural systems

Introduction

Brain-machine interfaces have reached an unprecedented capacity to measure and drive activity in the brain, allowing restoration of impaired sensory, cognitive or motor function. Classical control theory is pushed to its limit when aiming to design control laws that are suitable for large-scale, complex neural systems. This work proposes a scalable, data-driven, unified approach to study brain-machine-environment interaction using established tools from dynamical systems, optimal control theory, and deep learning.

Methods

To unify the methodology, we define the environment, neural system, and prosthesis in terms of differential equations with learnable parameters, which effectively reduce to recurrent neural networks in the discrete-time case. Drawing on tools from optimal control, we describe three ways to train the system: Direct optimization of an objective function, oracle-based learning, and reinforcement learning. These approaches are adapted to different assumptions about knowledge of system equations, linearity, differentiability, and observability.

Results

We apply the proposed framework to train an in-silico neural system to perform tasks in a linear and a nonlinear environment, namely particle stabilization and pole balancing. After training, this model is perturbed to simulate impairment of sensor and motor function. We show how a prosthetic controller can be trained to restore the behavior of the neural system under increasing levels of perturbation.

Discussion

We expect that the proposed framework will enable rapid and flexible synthesis of control algorithms for neural prostheses that reduce the need for in-vivo testing. We further highlight implications for sparse placement of prosthetic sensor and actuator components.

Leveraging deep learning to control neural oscillators

Modulation of the firing times of neural oscillators has long been an important control objective, with applications including Parkinson's disease, Tourette's syndrome, epilepsy, and learning. One common goal for such modulation is desynchronization, wherein two or more oscillators are stimulated to transition from firing in phase with each other to firing out of phase. The optimization of such stimuli has been well studied, but this typically relies on either a reduction of the dimensionality of the system or complete knowledge of the parameters and state of the system. This limits the applicability of results to real problems in neural control. Here, we present a trained artificial neural network capable of accurately estimating the effects of square-wave stimuli on neurons using minimal output information from the neuron. We then apply the results of this network to solve several related control problems in desynchronization, including desynchronizing pairs of neurons and achieving clustered subpopulations of neurons in the presence of coupling and noise.

Learning to control the brain through adaptive closed-loop patterned stimulation

OBJECTIVE

Stimulation of neural activity is an important scientific and clinical tool, causally testing hypotheses and treating neurodegenerative and neuropsychiatric diseases. However, current stimulation approaches cannot flexibly control the pattern of activity in populations of neurons. To address this, we developed a model-free, adaptive, closed-loop stimulation (ACLS) system that learns to use multi-site electrical stimulation to control the pattern of activity of a population of neurons.

APPROACH

The ACLS system combined multi-electrode electrophysiological recordings with multi-site electrical stimulation to simultaneously record the activity of a population of 5-15 multiunit neurons and deliver spatially-patterned electrical stimulation across 4-16 sites. Using a closed-loop learning system, ACLS iteratively updated the pattern of stimulation to reduce the difference between the observed neural response and a specific target pattern of firing rates in the recorded multiunits.

MAIN RESULTS

In silico and in vivo experiments showed ACLS learns to produce specific patterns of neural activity (in ∼15 min) and was robust to noise and drift in neural responses. In visual cortex of awake mice, ACLS learned electrical stimulation patterns that produced responses similar to the natural response evoked by visual stimuli. Similar to how repetition of a visual stimulus causes an adaptation in the neural response, the response to electrical stimulation was adapted when it was preceded by the associated visual stimulus.

SIGNIFICANCE

Our results show an ACLS system that can learn, in real-time, to generate specific patterns of neural activity. This work provides a framework for using model-free closed-loop learning to control neural activity.

Learning to Control Neurons using Aggregated Measurements

Controlling a population of neurons with one or a few control signals is challenging due to the severely underactuated nature of the control system and the inherent nonlinear dynamics of the neurons that are typically unknown. Control strategies that incorporate deep neural networks and machine learning techniques directly use data to learn a sequence of control actions for targeted manipulation of a population of neurons. However, these learning strategies inherently assume that perfect feedback data from each neuron at every sampling instant are available, and do not scale gracefully as the number of neurons in the population increases. As a result, the learning models need to be retrained whenever such a change occurs. In this work, we propose a learning strategy to design a control sequence by using population-level aggregated measurements and incorporate reinforcement learning techniques to find a (bounded, piecewise constant) control policy that fulfills the given control task. We demonstrate the feasibility of the proposed approach using numerical experiments on a finite population of nonlinear dynamical systems and canonical phase models that are widely used in neuroscience.

A Minimum Free Energy Model of Motor Learning

Even highly trained behaviors demonstrate variability, which is correlated with performance on current and future tasks. An objective of motor learning that is general enough to explain these phenomena has not been precisely formulated. In this six-week longitudinal learning study, participants practiced a set of motor sequences each day, and neuroimaging data were collected on days 1, 14, 28, and 42 to capture the neural correlates of the learning process. In our analysis, we first modeled the underlying neural and behavioral dynamics during learning. Our results demonstrate that the densities of whole-brain response, task-active regional response, and behavioral performance evolve according to a Fokker-Planck equation during the acquisition of a motor skill. We show that this implies that the brain concurrently optimizes the entropy of a joint density over neural response and behavior (as measured by sampling over multiple trials and subjects) and the expected performance under this density; we call this formulation of learning minimum free energy learning (MFEL). This model provides an explanation as to how behavioral variability can be tuned while simultaneously improving performance during learning. We then develop a novel variant of inverse reinforcement learning to retrieve the cost function optimized by the brain during the learning process, as well as the parameter used to tune variability. We show that this population-level analysis can be used to derive a learning objective that each subject optimizes during his or her study. In this way, MFEL effectively acts as a unifying principle, allowing users to precisely formulate learning objectives and infer their structure.