A Biologically Inspired Computational Model of Basal Ganglia in Action Selection

Valence-dependent dopaminergic modulation during reversal learning in Parkinson's disease: A neurocomputational approach

Reinforcement learning, crucial for behavior in dynamic environments, is driven by rewards and punishments, modulated by dopamine (DA) changes. This study explores the dopaminergic system's influence on learning, particularly in Parkinson's Disease (PD), where medication leads to impaired adaptability. Highlighting the role of tonic DA in signaling the valence of actions, this research investigates how DA affects response vigor and decision-making in PD. DA not only influences reward and punishment learning but also indicates the cognitive effort level and risk propensity in actions, which are essential for understanding and managing PD symptoms. In this work, we adapt our existing neurocomputational model of basal ganglia (BG) to simulate two reversal learning tasks proposed by Cools et al. We first optimized a Hebb rule for both probabilistic and deterministic reversal learning, conducted a sensitivity analysis (SA) on parameters related to DA effect, and compared performances between three groups: PD-ON, PD-OFF, and control subjects. In our deterministic task simulation, we explored switch error rates after unexpected task switches and found a U-shaped relationship between tonic DA levels and switch error frequency. Through SA, we classify these three groups. Then, assuming that the valence of the stimulus affects the tonic levels of DA, we were able to reproduce the results by Cools et al. As for the probabilistic task simulation, our results are in line with clinical data, showing similar trends with PD-ON, characterized by higher tonic DA levels that are correlated with increased difficulty in both acquisition and reversal tasks. Our study proposes a new hypothesis: valence, signaled by tonic DA levels, influences learning in PD, confirming the uncorrelation between phasic and tonic DA changes. This hypothesis challenges existing paradigms and opens new avenues for understanding cognitive processes in PD, particularly in reversal learning tasks.

3D printing combined with biopredictive dissolution and PBPK/PD modeling optimization and personalization of pharmacotherapy: Are we there yet?

Precision medicine requires selecting the appropriate dosage regimen for a patient using the right drug, at the right time. Model-Informed Precision Dosing (MIPD) is a concept suggesting utilization of model-based prediction methods for optimizing the treatment benefit-harm balance, based on individual characteristics of the patient, disease, treatment method, and other factors. Here, we discuss a theoretical workflow comprising several elements, beginning from the physiologically based pharmacokinetic/pharmacodynamic (PBPK/PD) models, through 3D printed tablets with the model proposed dose, information range and flow, and the patient themselves. We also describe each of these elements, and the connection between them, highlighting challenges and potential obstacles.

Feasibility of dopamine as a vector-valued feedback signal in the basal ganglia

It is well established that midbrain dopaminergic neurons support reinforcement learning (RL) in the basal ganglia by transmitting a reward prediction error (RPE) to the striatum. In particular, different computational models and experiments have shown that a striatum-wide RPE signal can support RL over a small discrete set of actions (e.g., no/no-go, choose left/right). However, there is accumulating evidence that the basal ganglia functions not as a selector between predefined actions but rather as a dynamical system with graded, continuous outputs. To reconcile this view with RL, there is a need to explain how dopamine could support learning of continuous outputs, rather than discrete action values. Inspired by the recent observations that besides RPE, the firing rates of midbrain dopaminergic neurons correlate with motor and cognitive variables, we propose a model in which dopamine signal in the striatum carries a vector-valued error feedback signal (a loss gradient) instead of a homogeneous scalar error (a loss). We implement a local, "three-factor" corticostriatal plasticity rule involving the presynaptic firing rate, a postsynaptic factor, and the unique dopamine concentration perceived by each striatal neuron. With this learning rule, we show that such a vector-valued feedback signal results in an increased capacity to learn a multidimensional series of real-valued outputs. Crucially, we demonstrate that this plasticity rule does not require precise nigrostriatal synapses but remains compatible with experimental observations of random placement of varicosities and diffuse volume transmission of dopamine.

Basal ganglia network dynamics and function: Role of direct, indirect and hyper-direct pathways in action selection

Basal ganglia (BG) are a widely recognized neural basis for action selection, but its decision-making mechanism is still a difficult problem for researchers. Therefore, we constructed a spiking neural network inspired by the BG anatomical data. Simulation experiments were based on the principle of dis-inhibition and our functional hypothesis within the BG: the direct pathway, the indirect pathway, and the hyper-direct pathway of the BG jointly implement the initiation execution and termination of motor programs. Firstly, we studied the dynamic process of action selection with the network, which contained intra-group competition and inter-group competition. Secondly, we focused on the effects of the stimulus intensity and the proportion of excitation and inhibition on the GPi/SNr. The results suggested that inhibition and excitation shape action selection. They also explained why the firing rate of GPi/SNr did not continue to increase in the action-selection experiment. Finally, we discussed the experimental results with the functional hypothesis. Uniquely, this paper summarized the decision-making neural mechanism of action selection based on the direct pathway, the indirect pathway, and the hyper-direct pathway within BG.

A mechanistic model of ADHD as resulting from dopamine phasic/tonic imbalance during reinforcement learning

Attention deficit hyperactivity disorder (ADHD) is the most common neurodevelopmental disorder in children. Although the involvement of dopamine in this disorder seems to be established, the nature of dopaminergic dysfunction remains controversial. The purpose of this study was to test whether the key response characteristics of ADHD could be simulated by a mechanistic model that combines a decrease in tonic dopaminergic activity with an increase in phasic responses in cortical-striatal loops during learning reinforcement. To this end, we combined a dynamic model of dopamine with a neurocomputational model of the basal ganglia with multiple action channels. We also included a dynamic model of tonic and phasic dopamine release and control, and a learning procedure driven by tonic and phasic dopamine levels. In the model, the dopamine imbalance is the result of impaired presynaptic regulation of dopamine at the terminal level. Using this model, virtual individuals from a dopamine imbalance group and a control group were trained to associate four stimuli with four actions with fully informative reinforcement feedback. In a second phase, they were tested without feedback. Subjects in the dopamine imbalance group showed poorer performance with more variable reaction times due to the presence of fast and very slow responses, difficulty in choosing between stimuli even when they were of high intensity, and greater sensitivity to noise. Learning history was also significantly more variable in the dopamine imbalance group, explaining 75% of the variability in reaction time using quadratic regression. The response profile of the virtual subjects varied as a function of the learning history variability index to produce increasingly severe impairment, beginning with an increase in response variability alone, then accumulating a decrease in performance and finally a learning deficit. Although ADHD is certainly a heterogeneous disorder, these results suggest that typical features of ADHD can be explained by a phasic/tonic imbalance in dopaminergic activity alone.

A model of working memory for encoding multiple items and ordered sequences exploiting the theta-gamma code

Recent experimental evidence suggests that oscillatory activity plays a pivotal role in the maintenance of information in working memory, both in rodents and humans. In particular, cross-frequency coupling between theta and gamma oscillations has been suggested as a core mechanism for multi-item memory. The aim of this work is to present an original neural network model, based on oscillating neural masses, to investigate mechanisms at the basis of working memory in different conditions. We show that this model, with different synapse values, can be used to address different problems, such as the reconstruction of an item from partial information, the maintenance of multiple items simultaneously in memory, without any sequential order, and the reconstruction of an ordered sequence starting from an initial cue. The model consists of four interconnected layers; synapses are trained using Hebbian and anti-Hebbian mechanisms, in order to synchronize features in the same items, and desynchronize features in different items. Simulations show that the trained network is able to desynchronize up to nine items without a fixed order using the gamma rhythm. Moreover, the network can replicate a sequence of items using a gamma rhythm nested inside a theta rhythm. The reduction in some parameters, mainly concerning the strength of GABAergic synapses, induce memory alterations which mimic neurological deficits. Finally, the network, isolated from the external environment (“imagination phase”) and stimulated with high uniform noise, can randomly recover sequences previously learned, and link them together by exploiting the similarity among items.

Phasic Dopamine Changes and Hebbian Mechanisms during Probabilistic Reversal Learning in Striatal Circuits: A Computational Study

Cognitive flexibility is essential to modify our behavior in a non-stationary environment and is often explored by reversal learning tasks. The basal ganglia (BG) dopaminergic system, under a top-down control of the pre-frontal cortex, is known to be involved in flexible action selection through reinforcement learning. However, how adaptive dopamine changes regulate this process and learning mechanisms for training the striatal synapses remain open questions. The current study uses a neurocomputational model of the BG, based on dopamine-dependent direct (Go) and indirect (NoGo) pathways, to investigate reinforcement learning in a probabilistic environment through a task that associates different stimuli to different actions. Here, we investigated: the efficacy of several versions of the Hebb rule, based on covariance between pre- and post-synaptic neurons, as well as the required control in phasic dopamine changes crucial to achieving a proper reversal learning. Furthermore, an original mechanism for modulating the phasic dopamine changes is proposed, assuming that the expected reward probability is coded by the activity of the winner Go neuron before a reward/punishment takes place. Simulations show that this original formulation for an automatic phasic dopamine control allows the achievement of a good flexible reversal even in difficult conditions. The current outcomes may contribute to understanding the mechanisms for active control of dopamine changes during flexible behavior. In perspective, it may be applied in neuropsychiatric or neurological disorders, such as Parkinson’s or schizophrenia, in which reinforcement learning is impaired.

Coarse-Grained Neural Network Model of the Basal Ganglia to Simulate Reinforcement Learning Tasks

Computational models of the basal ganglia (BG) provide a mechanistic account of different phenomena observed during reinforcement learning tasks performed by healthy individuals, as well as by patients with various nervous or mental disorders. The aim of the present work was to develop a BG model that could represent a good compromise between simplicity and completeness. Based on more complex (fine-grained neural network, FGNN) models, we developed a new (coarse-grained neural network, CGNN) model by replacing layers of neurons with single nodes that represent the collective behavior of a given layer while preserving the fundamental anatomical structures of BG. We then compared the functionality of both the FGNN and CGNN models with respect to several reinforcement learning tasks that are based on BG circuitry, such as the Probabilistic Selection Task, Probabilistic Reversal Learning Task and Instructed Probabilistic Selection Task. We showed that CGNN still has a functionality that mirrors the behavior of the most often used reinforcement learning tasks in human studies. The simplification of the CGNN model reduces its flexibility but improves the readability of the signal flow in comparison to more detailed FGNN models and, thus, can help to a greater extent in the translation between clinical neuroscience and computational modeling.

Response Systems, Antagonistic Responses, and the Behavioral Repertoire

While response systems are often mentioned in the behavioral and physiological literature, an explicit discussion of what response systems are is lacking. Here we argue that response systems can be understood as an interaction between anatomically constrained behavioral topographies occasioned by currently present stimuli and a history of reinforcement. “New” response systems can develop during the lifetime as the organism gains instrumental control of new fine-grained topographies. Within this framework, antagonistic responses compete within each response system based on environmental stimulation, and competition is resolved at the striatum-thalamo-cortical loops level. While response systems can be by definition independent from one another, separate systems are often recruited at the same time to engage in complex responses, which themselves may be selected by reinforcement as functional units.

Introduction to Focus Issue: Dynamical disease: A translational approach

The concept of Dynamical Diseases provides a framework to understand physiological control systems in pathological states due to their operating in an abnormal range of control parameters: this allows for the possibility of a return to normal condition by a redress of the values of the governing parameters. The analogy with bifurcations in dynamical systems opens the possibility of mathematically modeling clinical conditions and investigating possible parameter changes that lead to avoidance of their pathological states. Since its introduction, this concept has been applied to a number of physiological systems, most notably cardiac, hematological, and neurological. A quarter century after the inaugural meeting on dynamical diseases held in Mont Tremblant, Québec [Bélair et al., Dynamical Diseases: Mathematical Analysis of Human Illness (American Institute of Physics, Woodbury, NY, 1995)], this Focus Issue offers an opportunity to reflect on the evolution of the field in traditional areas as well as contemporary data-based methods.

An integrative model of Parkinson's disease treatment including levodopa pharmacokinetics, dopamine kinetics, basal ganglia neurotransmission and motor action throughout disease progression

Levodopa is considered the gold standard treatment of Parkinson's disease. Although very effective in alleviating symptoms at their onset, its chronic use with the progressive neuronal denervation in the basal ganglia leads to a decrease in levodopa's effect duration and to the appearance of motor complications. This evolution challenges the establishment of optimal regimens to manage the symptoms as the disease progresses. Based on up-to-date pathophysiological and pharmacological knowledge, we developed an integrative model for Parkinson's disease to evaluate motor function in response to levodopa treatment as the disease progresses. We combined a pharmacokinetic model of levodopa to a model of dopamine's kinetics and a neurocomputational model of basal ganglia. The parameter values were either measured directly or estimated from human and animal data. The concentrations and behaviors predicted by our model were compared to available information and data. Using this model, we were able to predict levodopa plasma concentration, its related dopamine concentration in the brain and the response performance of a motor task for different stages of disease.

A non-linear deterministic model of action selection in the basal ganglia to simulate motor fluctuations in Parkinson's disease

Motor fluctuations and dyskinesias are severe complications of Parkinson's disease (PD), especially evident at its advanced stage, under long-term levodopa therapy. Despite their strong clinical prevalence, the neural origin of these motor symptoms is still a subject of intense debate. In this work, a non-linear deterministic neurocomputational model of the basal ganglia (BG), inspired by biology, is used to provide more insights into possible neural mechanisms at the basis of motor complications in PD. In particular, the model is used to simulate the finger tapping task. The model describes the main neural pathways involved in the BG to select actions [the direct or Go, the indirect or NoGo, and the hyperdirect pathways via the action of the sub-thalamic nucleus (STN)]. A sensitivity analysis is performed on some crucial model parameters (the dopamine level, the strength of the STN mechanism, and the strength of competition among different actions in the motor cortex) at different levels of synapses, reflecting major or minor motor training. Depending on model parameters, results show that the model can reproduce a variety of clinically relevant motor patterns, including normokinesia, bradykinesia, several attempts before movement, freezing, repetition, and also irregular fluctuations. Motor symptoms are, especially, evident at low or high dopamine levels, with excessive strength of the STN and with weak competition among alternative actions. Moreover, these symptoms worsen if the synapses are subject to insufficient learning. The model may help improve the comprehension of motor complications in PD and, ultimately, may contribute to the treatment design.

Mathematical modeling and parameter estimation of levodopa motor response in patients with parkinson disease

Parkinson disease (PD) is characterized by a clear beneficial motor response to levodopa (LD) treatment. However, with disease progression and longer LD exposure, drug-related motor fluctuations usually occur. Recognition of the individual relationship between LD concentration and its effect may be difficult, due to the complexity and variability of the mechanisms involved. This work proposes an innovative procedure for the automatic estimation of LD pharmacokinetics and pharmacodynamics parameters, by a biologically-inspired mathematical model. An original issue, compared with previous similar studies, is that the model comprises not only a compartmental description of LD pharmacokinetics in plasma and its effect on the striatal neurons, but also a neurocomputational model of basal ganglia action selection. Parameter estimation was achieved on 26 patients (13 with stable and 13 with fluctuating LD response) to mimic plasma LD concentration and alternate finger tapping frequency along four hours after LD administration, automatically minimizing a cost function of the difference between simulated and clinical data points. Results show that individual data can be satisfactorily simulated in all patients and that significant differences exist in the estimated parameters between the two groups. Specifically, the drug removal rate from the effect compartment, and the Hill coefficient of the concentration-effect relationship were significantly higher in the fluctuating than in the stable group. The model, with individualized parameters, may be used to reach a deeper comprehension of the PD mechanisms, mimic the effect of medication, and, based on the predicted neural responses, plan the correct management and design innovative therapeutic procedures.

A Brain-Inspired Decision-Making Spiking Neural Network and Its Application in Unmanned Aerial Vehicle

Decision-making is a crucial cognitive function for various animal species surviving in nature, and it is also a fundamental ability for intelligent agents. To make a step forward in the understanding of the computational mechanism of human-like decision-making, this paper proposes a brain-inspired decision-making spiking neural network (BDM-SNN) and applies it to decision-making tasks on intelligent agents. This paper makes the following contributions: (1) A spiking neural network (SNN) is used to model human decision-making neural circuit from both connectome and functional perspectives. (2) The proposed model combines dopamine and spike-timing-dependent plasticity (STDP) mechanisms to modulate the network learning process, which indicates more biological inspiration. (3) The model considers the effects of interactions among sub-areas in PFC on accelerating the learning process. (4) The proposed model can be easily applied to decision-making tasks in intelligent agents, such as an unmanned aerial vehicle (UAV) flying through a window and a UAV avoiding an obstacle. The experimental results support the effectiveness of the model. Compared with traditional reinforcement learning and existing biologically inspired methods, our method contains more biologically-inspired mechanistic principles, has greater accuracy and is faster.

A Mathematical Model of Levodopa Medication Effect on Basal Ganglia in Parkinson's Disease: An Application to the Alternate Finger Tapping Task

Malfunctions in the neural circuitry of the basal ganglia (BG), induced by alterations in the dopaminergic system, are responsible for an array of motor disorders and milder cognitive issues in Parkinson's disease (PD). Recently Baston and Ursino (2015a) presented a new neuroscience mathematical model aimed at exploring the role of basal ganglia in action selection. The model is biologically inspired and reproduces the main BG structures and pathways, modeling explicitly both the dopaminergic and the cholinergic system. The present work aims at interfacing this neurocomputational model with a compartmental model of levodopa, to propose a general model of medicated Parkinson's disease. Levodopa effect on the striatum was simulated with a two-compartment model of pharmacokinetics in plasma joined with a motor effect compartment. The latter is characterized by the levodopa removal rate and by a sigmoidal relationship (Hill law) between concentration and effect. The main parameters of this relationship are saturation, steepness, and the half-maximum concentration. The effect of levodopa is then summed to a term representing the endogenous dopamine effect, and is used as an external input for the neurocomputation model; this allows both the temporal aspects of medication and the individual patient characteristics to be simulated. The frequency of alternate tapping is then used as the outcome of the whole model, to simulate effective clinical scores. Pharmacokinetic-pharmacodynamic modeling was preliminary performed on data of six patients with Parkinson's disease (both "stable" and "wearing-off" responders) after levodopa standardized oral dosing over 4 h. Results show that the model is able to reproduce the temporal profiles of levodopa in plasma and the finger tapping frequency in all patients, discriminating between different patterns of levodopa motor response. The more influential parameters are the Hill coefficient, related with the slope of the effect sigmoidal relationship, the drug concentration at half-maximum effect, and the drug removal rate from the effect compartment. The model can be of value to gain a deeper understanding on the pharmacokinetics and pharmacodynamics of the medication, and on the way dopamine is exploited in the neural circuitry of the basal ganglia in patients at different stages of the disease progression.

Effects of Levodopa on Postural Strategies in Parkinson's disease

Altered postural control and balance are major disabling issues of Parkinson's disease (PD). Static and dynamic posturography have provided insight into PD's postural deficits; however, little is known about impairments in postural coordination. We hypothesized that subjects with PD would show more ankle strategy during quiet stance than healthy control subjects, who would include some hip strategy, and this stiffer postural strategy would increase with disease progression. We quantified postural strategy and sway dispersion with inertial sensors (one placed on the shank and one on the posterior trunk at L5 level) while subjects were standing still with their eyes open. A total of 70 subjects with PD, including a mild group (H&Y≤2, N=33) and a more severe group (H&Y≥3, N=37), were assessed while OFF and while ON levodopa medication. We also included a healthy control group (N=21). Results showed an overall preference of ankle strategy in all groups while maintaining balance. Postural strategy was significantly lower ON compared to OFF medication (indicating more hip strategy), but no effect of disease stage was found. Instead, sway dispersion was significantly larger in ON compared to OFF medication, and significantly larger in the more severe PD group compared to the mild. In addition, increased hip strategy during stance was associated with poorer self-perception of balance.