Event-based minimum-time control of oscillatory neuron models: phase randomization, maximal spike rate increase, and desynchronization

Anti-control of periodic firing in HR model in the aspects of position, amplitude and frequency

This paper proposes a novel controller to control position, amplitude and frequency of periodic firing activity in Hindmarsh-Rose model based on Hopf bifurcation theory which is composed of linear control gain and nonlinear control gain. First, we select the activation of the fast ion channel as control parameter. Based on explicit criterion of Hopf bifurcation, a series of conditions are obtained to derive the linear gains of controller responsible for control of the location where the periodic firing activity occurs. Then, based on the control parameter, a series of conditions are obtained to derive the nonlinear gains of controller responsible for controlling the amplitude and frequency of periodic firing activity by using center manifold and normal form. Finally, the numerical experiments show that our controller can make the periodic firing activity occur at designed value and control the amplitude and frequency of periodic firing activity by adjusting nonlinear control gain of controller.

Learning to control the brain through adaptive closed-loop patterned stimulation

OBJECTIVE

Stimulation of neural activity is an important scientific and clinical tool, causally testing hypotheses and treating neurodegenerative and neuropsychiatric diseases. However, current stimulation approaches cannot flexibly control the pattern of activity in populations of neurons. To address this, we developed a model-free, adaptive, closed-loop stimulation (ACLS) system that learns to use multi-site electrical stimulation to control the pattern of activity of a population of neurons.

APPROACH

The ACLS system combined multi-electrode electrophysiological recordings with multi-site electrical stimulation to simultaneously record the activity of a population of 5-15 multiunit neurons and deliver spatially-patterned electrical stimulation across 4-16 sites. Using a closed-loop learning system, ACLS iteratively updated the pattern of stimulation to reduce the difference between the observed neural response and a specific target pattern of firing rates in the recorded multiunits.

MAIN RESULTS

In silico and in vivo experiments showed ACLS learns to produce specific patterns of neural activity (in ∼15 min) and was robust to noise and drift in neural responses. In visual cortex of awake mice, ACLS learned electrical stimulation patterns that produced responses similar to the natural response evoked by visual stimuli. Similar to how repetition of a visual stimulus causes an adaptation in the neural response, the response to electrical stimulation was adapted when it was preceded by the associated visual stimulus.

SIGNIFICANCE

Our results show an ACLS system that can learn, in real-time, to generate specific patterns of neural activity. This work provides a framework for using model-free closed-loop learning to control neural activity.

Phase reduction and phase-based optimal control for biological systems: a tutorial

A powerful technique for the analysis of nonlinear oscillators is the rigorous reduction to phase models, with a single variable describing the phase of the oscillation with respect to some reference state. An analog to phase reduction has recently been proposed for systems with a stable fixed point, and phase reduction for periodic orbits has recently been extended to take into account transverse directions and higher-order terms. This tutorial gives a unified treatment of such phase reduction techniques and illustrates their use through mathematical and biological examples. It also covers the use of phase reduction for designing control algorithms which optimally change properties of the system, such as the phase of the oscillation. The control techniques are illustrated for example neural and cardiac systems.

Control of neural systems at multiple scales using model-free, deep reinforcement learning

Recent improvements in hardware and data collection have lowered the barrier to practical neural control. Most of the current contributions to the field have focus on model-based control, however, models of neural systems are quite complex and difficult to design. To circumvent these issues, we adapt a model-free method from the reinforcement learning literature, Deep Deterministic Policy Gradients (DDPG). Model-free reinforcement learning presents an attractive framework because of the flexibility it offers, allowing the user to avoid modeling system dynamics. We make use of this feature by applying DDPG to models of low-level and high-level neural dynamics. We show that while model-free, DDPG is able to solve more difficult problems than can be solved by current methods. These problems include the induction of global synchrony by entrainment of weakly coupled oscillators and the control of trajectories through a latent phase space of an underactuated network of neurons. While this work has been performed on simulated systems, it suggests that advances in modern reinforcement learning may enable the solution of fundamental problems in neural control and movement towards more complex objectives in real systems.

Phase model-based neuron stabilization into arbitrary clusters

Deep brain stimulation (DBS) is a common method of combating pathological conditions associated with Parkinson's disease, Tourette syndrome, essential tremor, and other disorders, but whose mechanisms are not fully understood. One hypothesis, supported experimentally, is that some symptoms of these disorders are associated with pathological synchronization of neurons in the basal ganglia and thalamus. For this reason, there has been interest in recent years in finding efficient ways to desynchronize neurons that are both fast-acting and low-power. Recent results on coordinated reset and periodically forced oscillators suggest that forming distinct clusters of neurons may prove to be more effective than achieving complete desynchronization, in particular by promoting plasticity effects that might persist after stimulation is turned off. Current proposed methods for achieving clustering frequently require either multiple input sources or precomputing the control signal. We propose here a control strategy for clustering, based on an analysis of the reduced phase model for a set of identical neurons, that allows for real-time, single-input control of a population of neurons with low-amplitude, low total energy signals. After demonstrating its effectiveness on phase models, we apply it to full state models to demonstrate its validity. We also discuss the effects of coupling on the efficacy of the strategy proposed and demonstrate that the clustering can still be accomplished in the presence of weak to moderate electrotonic coupling.

A new approach to optimal control of conductance-based spiking neurons

This paper presents an algorithm for solving the minimum-energy optimal control problem of conductance-based spiking neurons. The basic procedure is (1) to construct a conductance-based spiking neuron oscillator as an affine nonlinear system, (2) to formulate the optimal control problem of the affine nonlinear system as a boundary value problem based on Pontryagin's maximum principle, and (3) to solve the boundary value problem using the homotopy perturbation method. The construction of the minimum-energy optimal control in the framework of the homotopy perturbation technique is novel and valid for a broad class of nonlinear conductance-based neuron models. The applicability of our method in the FitzHugh-Nagumo and Hindmarsh-Rose models is validated by simulations.

How to desynchronize quorum-sensing networks

In this paper we investigate how so-called quorum-sensing networks can be desynchronized. Such networks, which arise in many important application fields, such as systems biology, are characterized by the fact that direct communication between network nodes is superimposed to communication with a shared, environmental variable. In particular, we provide a new sufficient condition ensuring that the trajectories of these quorum-sensing networks diverge from their synchronous evolution. Then, we apply our result to study two applications.

Fast phase randomization via two-folds

A two-fold is a singular point on the discontinuity surface of a piecewise-smooth vector field, at which the vector field is tangent to the discontinuity surface on both sides. If an orbit passes through an invisible two-fold (also known as a Teixeira singularity) before settling to regular periodic motion, then the phase of that motion cannot be determined from initial conditions, and, in the presence of small noise, the asymptotic phase of a large number of sample solutions is highly random. In this paper, we show how the probability distribution of the asymptotic phase depends on the global nonlinear dynamics. We also show how the phase of a smooth oscillator can be randomized by applying a simple discontinuous control law that generates an invisible two-fold. We propose that such a control law can be used to desynchronize a collection of oscillators, and that this manner of phase randomization is fast compared with existing methods (which use fixed points as phase singularities), because there is no slowing of the dynamics near a two-fold.

Closed-loop and activity-guided optogenetic control

Advances in optical manipulation and observation of neural activity have set the stage for widespread implementation of closed-loop and activity-guided optical control of neural circuit dynamics. Closing the loop optogenetically (i.e., basing optogenetic stimulation on simultaneously observed dynamics in a principled way) is a powerful strategy for causal investigation of neural circuitry. In particular, observing and feeding back the effects of circuit interventions on physiologically relevant timescales is valuable for directly testing whether inferred models of dynamics, connectivity, and causation are accurate in vivo. Here we highlight technical and theoretical foundations as well as recent advances and opportunities in this area, and we review in detail the known caveats and limitations of optogenetic experimentation in the context of addressing these challenges with closed-loop optogenetic control in behaving animals.

Control of abnormal synchronization in neurological disorders

In the nervous system, synchronization processes play an important role, e.g., in the context of information processing and motor control. However, pathological, excessive synchronization may strongly impair brain function and is a hallmark of several neurological disorders. This focused review addresses the question of how an abnormal neuronal synchronization can specifically be counteracted by invasive and non-invasive brain stimulation as, for instance, by deep brain stimulation for the treatment of Parkinson’s disease, or by acoustic stimulation for the treatment of tinnitus. On the example of coordinated reset (CR) neuromodulation, we illustrate how insights into the dynamics of complex systems contribute to successful model-based approaches, which use methods from synergetics, non-linear dynamics, and statistical physics, for the development of novel therapies for normalization of brain function and synaptic connectivity. Based on the intrinsic multistability of the neuronal populations induced by spike timing-dependent plasticity (STDP), CR neuromodulation utilizes the mutual interdependence between synaptic connectivity and dynamics of the neuronal networks in order to restore more physiological patterns of connectivity via desynchronization of neuronal activity. The very goal is to shift the neuronal population by stimulation from an abnormally coupled and synchronized state to a desynchronized regime with normalized synaptic connectivity, which significantly outlasts the stimulation cessation, so that long-lasting therapeutic effects can be achieved.

Locally optimal extracellular stimulation for chaotic desynchronization of neural populations

We use optimal control theory to design a methodology to find locally optimal stimuli for desynchronization of a model of neurons with extracellular stimulation. This methodology yields stimuli which lead to positive Lyapunov exponents, and hence desynchronizes a neural population. We analyze this methodology in the presence of interneuron coupling to make predictions about the strength of stimulation required to overcome synchronizing effects of coupling. This methodology suggests a powerful alternative to pulsatile stimuli for deep brain stimulation as it uses less energy than pulsatile stimuli, and could eliminate the time consuming tuning process.

Stochastic optimal control of single neuron spike trains

OBJECTIVE

External control of spike times in single neurons can reveal important information about a neuron's sub-threshold dynamics that lead to spiking, and has the potential to improve brain-machine interfaces and neural prostheses. The goal of this paper is the design of optimal electrical stimulation of a neuron to achieve a target spike train under the physiological constraint to not damage tissue.

APPROACH

We pose a stochastic optimal control problem to precisely specify the spike times in a leaky integrate-and-fire (LIF) model of a neuron with noise assumed to be of intrinsic or synaptic origin. In particular, we allow for the noise to be of arbitrary intensity. The optimal control problem is solved using dynamic programming when the controller has access to the voltage (closed-loop control), and using a maximum principle for the transition density when the controller only has access to the spike times (open-loop control).

MAIN RESULTS

We have developed a stochastic optimal control algorithm to obtain precise spike times. It is applicable in both the supra-threshold and sub-threshold regimes, under open-loop and closed-loop conditions and with an arbitrary noise intensity; the accuracy of control degrades with increasing intensity of the noise. Simulations show that our algorithms produce the desired results for the LIF model, but also for the case where the neuron dynamics are given by more complex models than the LIF model. This is illustrated explicitly using the Morris-Lecar spiking neuron model, for which an LIF approximation is first obtained from a spike sequence using a previously published method. We further show that a related control strategy based on the assumption that there is no noise performs poorly in comparison to our noise-based strategies. The algorithms are numerically intensive and may require efficiency refinements to achieve real-time control; in particular, the open-loop context is more numerically demanding than the closed-loop one.

SIGNIFICANCE

Our main contribution is the online feedback control of a noisy neuron through modulation of the input, taking into account physiological constraints on the control. A precise and robust targeting of neural activity based on stochastic optimal control has great potential for regulating neural activity in e.g. prosthetic applications and to improve our understanding of the basic mechanisms by which neuronal firing patterns can be controlled in vivo.

Closing the loop of deep brain stimulation

High-frequency deep brain stimulation is used to treat a wide range of brain disorders, like Parkinson's disease. The stimulated networks usually share common electrophysiological signatures, including hyperactivity and/or dysrhythmia. From a clinical perspective, HFS is expected to alleviate clinical signs without generating adverse effects. Here, we consider whether the classical open-loop HFS fulfills these criteria and outline current experimental or theoretical research on the different types of closed-loop DBS that could provide better clinical outcomes. In the first part of the review, the two routes followed by HFS-evoked axonal spikes are explored. In one direction, orthodromic spikes functionally de-afferent the stimulated nucleus from its downstream target networks. In the opposite direction, antidromic spikes prevent this nucleus from being influenced by its afferent networks. As a result, the pathological synchronized activity no longer propagates from the cortical networks to the stimulated nucleus. The overall result can be described as a reversible functional de-afferentation of the stimulated nucleus from its upstream and downstream nuclei. In the second part of the review, the latest advances in closed-loop DBS are considered. Some of the proposed approaches are based on mathematical models, which emphasize different aspects of the parkinsonian basal ganglia: excessive synchronization, abnormal firing-rate rhythms, and a deficient thalamo-cortical relay. The stimulation strategies are classified depending on the control-theory techniques on which they are based: adaptive and on-demand stimulation schemes, delayed and multi-site approaches, stimulations based on proportional and/or derivative control actions, optimal control strategies. Some of these strategies have been validated experimentally, but there is still a large reservoir of theoretical work that may point to ways of improving practical treatment.

Minimum energy control for in vitro neurons

OBJECTIVE

To demonstrate the applicability of optimal control theory for designing minimum energy charge-balanced input waveforms for single periodically-firing in vitro neurons from brain slices of Long-Evans rats.

APPROACH

The method of control uses the phase model of a neuron and does not require prior knowledge of the neuron's biological details. The phase model of a neuron is a one-dimensional model that is characterized by the neuron's phase response curve (PRC), a sensitivity measure of the neuron to a stimulus applied at different points in its firing cycle. The PRC for each neuron is experimentally obtained by measuring the shift in phase due to a short-duration pulse injected into the periodically-firing neuron at various phase values. Based on the measured PRC, continuous-time, charge-balanced, minimum energy control waveforms have been designed to regulate the next firing time of the neuron upon application at the onset of an action potential.

MAIN RESULT

The designed waveforms can achieve the inter-spike-interval regulation for in vitro neurons with energy levels that are lower than those of conventional monophasic pulsatile inputs of past studies by at least an order of magnitude. They also provide the advantage of being charge-balanced. The energy efficiency of these waveforms is also shown by performing several supporting simulations that compare the performance of the designed waveforms against that of phase shuffled surrogate inputs, variants of the minimum energy waveforms obtained from suboptimal PRCs, as well as pulsatile stimuli that are applied at the point of maximum PRC. It was found that the minimum energy waveforms perform better than all other stimuli both in terms of control and in the amount of energy used. Specifically, it was seen that these charge-balanced waveforms use at least an order of magnitude less energy than conventional monophasic pulsatile stimuli.

SIGNIFICANCE

The significance of this work is that it uses concepts from the theory of optimal control and introduces a novel approach in designing minimum energy charge-balanced input waveforms for neurons that are robust to noise and implementable in electrophysiological experiments.

Control strategies for underactuated neural ensembles driven by optogenetic stimulation

Motivated by experiments employing optogenetic stimulation of cortical regions, we consider spike control strategies for ensembles of uncoupled integrate and fire neurons with a common conductance input. We construct strategies for control of spike patterns, that is, multineuron trains of action potentials, up to some maximal spike rate determined by the neural biophysics. We emphasize a constructive role for parameter heterogeneity, and find a simple rule for controllability in pairs of neurons. In particular, we determine parameters for which common drive is not limited to inducing synchronous spiking. For large ensembles, we determine how the number of controllable neurons varies with the number of observed (recorded) neurons, and what collateral spiking occurs in the full ensemble during control of the subensemble. While complete control of spiking in every neuron is not possible with a single input, we find that a degree of subensemble control is made possible by exploiting dynamical heterogeneity. As most available technologies for neural stimulation are underactuated, in the sense that the number of target neurons far exceeds the number of independent channels of stimulation, these results suggest partial control strategies that may be important in the development of sensory neuroprosthetics and other neurocontrol applications.

Minimum energy desynchronizing control for coupled neurons

We employ optimal control theory to design an event-based, minimum energy, desynchronizing control stimulus for a network of pathologically synchronized, heterogeneously coupled neurons. This works by optimally driving the neurons to their phaseless sets, switching the control off, and letting the phases of the neurons randomize under intrinsic background noise. An event-based minimum energy input may be clinically desirable for deep brain stimulation treatment of neurological diseases, like Parkinson's disease. The event-based nature of the input results in its administration only when it is necessary, which, in general, amounts to fewer applications, and hence, less charge transfer to and from the tissue. The minimum energy nature of the input may also help prolong battery life for implanted stimulus generators. For the example considered, it is shown that the proposed control causes a considerable amount of randomization in the timing of each neuron's next spike, leading to desynchronization for the network.

Controlling spike timing and synchrony in oscillatory neurons

We describe an algorithm to control synchrony between two periodically firing neurons. The control scheme operates in real-time using a dynamic clamp platform. This algorithm is a low-impact stimulation method that brings the neurons toward the desired level of synchrony over the course of several neuron firing periods. As a proof of principle, we demonstrate the versatility of the algorithm using real-time conductance models and then show its performance with biological neurons of hippocampal region CA1 and entorhinal cortex.

Time optimal control of spiking neurons

By injecting an electrical current control stimulus into a neuron, one can change its inter-spike intervals. In this paper, we investigate the time optimal control problem for periodically firing neurons, represented by different one-dimensional phase models, and find analytical expressions for the minimum and maximum values of inter-spike intervals achievable with small bounded control stimuli. We consider two cases: with a charge-balance constraint on the input, and without it. The analytical calculations are supported with numerical results for examples of qualitatively different neuron models.

Exploring optimal current stimuli that provide membrane voltage tracking in a neuron model

Studying neurons from an energy efficiency perspective has produced results in the research literature. This paper presents a method that enables computation of low energy input current stimuli that are able to drive a reduced Hodgkin-Huxley neuron model to approximate a prescribed time-varying reference membrane voltage. An optimal control technique is used to discover an input current that optimally minimizes a user selected balance between the square of the input stimulus current (input current 'energy') and the difference between the reference voltage and the membrane voltage (tracking error) over a stimulation period. Selecting reference signals to be membrane voltages produced by the neuron model in response to common types of input currents i(t) enables a comparison between i(t) and the determined optimal current stimulus i*(t). The intent is not to modify neuron dynamics, but through comparison of i(t) and i*(t) provide insight into neuron dynamics. Simulation results for four different bifurcation types demonstrate that this method consistently finds lower energy stimulus currents i*(t) that are able to approximate membrane voltages as produced by higher energy input currents i(t) in this neuron model.