A spiking Basal Ganglia model of synchrony, exploration and decision making

A Computational Model of Deep Brain Stimulation for Parkinson's Disease Tremor and Bradykinesia

Parkinson's disease (PD) is a progressive neurological disorder that is typically characterized by a range of motor dysfunctions, and its impact extends beyond physical abnormalities into emotional well-being and cognitive symptoms. The loss of dopaminergic neurons in the substantia nigra pars compacta (SNc) leads to an array of dysfunctions in the functioning of the basal ganglia (BG) circuitry that manifests into PD. While active research is being carried out to find the root cause of SNc cell death, various therapeutic techniques are used to manage the symptoms of PD. The most common approach in managing the symptoms is replenishing the lost dopamine in the form of taking dopaminergic medications such as levodopa, despite its long-term complications. Another commonly used intervention for PD is deep brain stimulation (DBS). DBS is most commonly used when levodopa medication efficacy is reduced, and, in combination with levodopa medication, it helps reduce the required dosage of medication, prolonging the therapeutic effect. DBS is also a first choice option when motor complications such as dyskinesia emerge as a side effect of medication. Several studies have also reported that though DBS is found to be effective in suppressing severe motor symptoms such as tremors and rigidity, it has an adverse effect on cognitive capabilities. Henceforth, it is important to understand the exact mechanism of DBS in alleviating motor symptoms. A computational model of DBS stimulation for motor symptoms will offer great insights into understanding the mechanisms underlying DBS, and, along this line, in our current study, we modeled a cortico-basal ganglia circuitry of arm reaching, where we simulated healthy control (HC) and PD symptoms as well as the DBS effect on PD tremor and bradykinesia. Our modeling results reveal that PD tremors are more correlated with the theta band, while bradykinesia is more correlated with the beta band of the frequency spectrum of the local field potential (LFP) of the subthalamic nucleus (STN) neurons. With a DBS current of 220 pA, 130 Hz, and a 100 microsecond pulse-width, we could found the maximum therapeutic effect for the pathological dynamics simulated using our model using a set of parameter values. However, the exact DBS characteristics vary from patient to patient, and this can be further studied by exploring the model parameter space. This model can be extended to study different DBS targets and accommodate cognitive dynamics in the future to study the impact of DBS on cognitive symptoms and thereby optimize the parameters to produce optimal performance effects across modalities. Combining DBS with rehabilitation is another frontier where DBS can reduce symptoms such as tremors and rigidity, enabling patients to participate in their therapy. With DBS providing instant relief to patients, a combination of DBS and rehabilitation can enhance neural plasticity. One of the key motivations behind combining DBS with rehabilitation is to expect comparable results in motor performance even with milder DBS currents.

Improved control effect of pathological oscillations by using delayed feedback stimulation in neural mass model with pedunculopontine nucleus

BACKGROUND

The role of delayed feedback stimulation in the discussion of Parkinson's disease (PD) has recently received increasing attention. Stimulation of pedunculopontine nucleus (PPN) is an emerging treatment for PD. However, the effect of PPN in regulating PD is ignored, and the delayed feedback stimulation algorithm is facing some problems in parameter selection.

METHODS

On the basis of a neural mass model, we established a new network for PPN. Four types of delayed feedback stimulation schemes were designed, such as stimulating subthalamic nucleus (STN) with the local field potentials (LFPs) of STN nucleus, globus pallidus (GPe) with the LFPs of Gpe nucleus, PPN with the LFPs of Gpe nucleus, and STN with the LFPs of PPN nucleus.

RESULTS

In this study, we found that all four kinds of delayed feedback schemes are effective, suggesting that the algorithm is simple and more effective in experiments. More specifically, the other three control schemes improved the control performance and reduced the stimulation energy expenditure compared with traditional stimulating STN itself only.

CONCLUSION

PPN stimulation can affect the new network and help to suppress pathological oscillations for each neuron. We hope that our results can gain an insight into the future clinical treatment.

A generalized reinforcement learning based deep neural network agent model for diverse cognitive constructs

Human cognition is characterized by a wide range of capabilities including goal-oriented selective attention, distractor suppression, decision making, response inhibition, and working memory. Much research has focused on studying these individual components of cognition in isolation, whereas in several translational applications for cognitive impairment, multiple cognitive functions are altered in a given individual. Hence it is important to study multiple cognitive abilities in the same subject or, in computational terms, model them using a single model. To this end, we propose a unified, reinforcement learning-based agent model comprising of systems for representation, memory, value computation and exploration. We successfully modeled the aforementioned cognitive tasks and show how individual performance can be mapped to model meta-parameters. This model has the potential to serve as a proxy for cognitively impaired conditions, and can be used as a clinical testbench on which therapeutic interventions can be simulated first before delivering to human subjects.

BiCoSS: Toward Large-Scale Cognition Brain With Multigranular Neuromorphic Architecture

The further exploration of the neural mechanisms underlying the biological activities of the human brain depends on the development of large-scale spiking neural networks (SNNs) with different categories at different levels, as well as the corresponding computing platforms. Neuromorphic engineering provides approaches to high-performance biologically plausible computational paradigms inspired by neural systems. In this article, we present a biological-inspired cognitive supercomputing system (BiCoSS) that integrates multiple granules (GRs) of SNNs to realize a hybrid compatible neuromorphic platform. A scalable hierarchical heterogeneous multicore architecture is presented, and a synergistic routing scheme for hybrid neural information is proposed. The BiCoSS system can accommodate different levels of GRs and biological plausibility of SNN models in an efficient and scalable manner. Over four million neurons can be realized on BiCoSS with a power efficiency of 2.8k larger than the GPU platform, and the average latency of BiCoSS is 3.62 and 2.49 times higher than conventional architectures of digital neuromorphic systems. For the verification, BiCoSS is used to replicate various biological cognitive activities, including motor learning, action selection, context-dependent learning, and movement disorders. Comprehensively considering the programmability, biological plausibility, learning capability, computational power, and scalability, BiCoSS is shown to outperform the alternative state-of-the-art works for large-scale SNN, while its real-time computational capability enables a wide range of potential applications.

A systematic review of neural, cognitive, and clinical studies of anger and aggression

Anger and aggression have large impact on people's safety and the society at large. In order to provide an intervention to minimise aggressive behaviours, it is important to understand the neural and cognitive aspects of anger and aggression. In this systematic review, we investigate the cognitive and neural aspects of anger-related processes, including anger-related behaviours and anger reduction. Using this information, we then review prior existing methods on the treatment of anger-related disorders as well as anger management, including mindfulness and cognitive behavioural therapy. At the cognitive level, our review that anger is associated with excessive attention to anger-related stimuli and impulsivity. At the neural level, anger is associated with abnormal functioning of the amygdala and ventromedial prefrontal cortex. In conclusions, based on cognitive and neural studies, we here argue that mindfulness based cognitive behavioural therapy may be better at reducing anger and aggression than other behavioural treatments, such as cognitive behavioural therapy or mindfulness alone. We provide key information on future research work and best ways to manage anger and reduce aggression. Importantly, future research should investigate how anger related behaviours is acquired and how stress impacts the development of anger.

A neurocomputational model of creative processes

Creativity is associated with finding novel, surprising, and useful solutions. We argue that creative cognitive processes, divergent thinking, abstraction, and improvisation are constructed on different novelty-based processes. The prefrontal cortex plays a role in creative ideation by providing a control mechanism. Moreover, thinking about novel solutions activates the distant or loosely connected neurons of a semantic network that involves the hippocampus. Novelty can also be interpreted as different combinations of earlier learned processes, such as the motor sequencing mechanism of the basal ganglia. In addition, the cerebellum is responsible for the precise control of movements, which is particularly important in improvisation. Our neurocomputational perspective is based on three creative processes centered on novelty seeking, subserved by the prefrontal cortex, hippocampus, cerebellum, basal ganglia, and dopamine. The algorithmic implementation of our model would enable us to describe commonalities and differences between these creative processes based on the proposed neural circuitry. Given that most previous studies have mainly provided theoretical and conceptual models of creativity, this article presents the first brain-inspired neural network model of creative cognition.

A Multi-Scale Computational Model of Levodopa-Induced Toxicity in Parkinson's Disease

Parkinson's disease (PD) is caused by the progressive loss of dopaminergic cells in substantia nigra pars compacta (SNc). The root cause of this cell loss in PD is still not decisively elucidated. A recent line of thinking has traced the cause of PD neurodegeneration to metabolic deficiency. Levodopa (L-DOPA), a precursor of dopamine, used as a symptom-relieving treatment for PD, leads to positive and negative outcomes. Several researchers inferred that L-DOPA might be harmful to SNc cells due to oxidative stress. The role of L-DOPA in the course of the PD pathogenesis is still debatable. We hypothesize that energy deficiency can lead to L-DOPA-induced toxicity in two ways: by promoting dopamine-induced oxidative stress and by exacerbating excitotoxicity in SNc. We present a systems-level computational model of SNc-striatum, which will help us understand the mechanism behind neurodegeneration postulated above and provide insights into developing disease-modifying therapeutics. It was observed that SNc terminals are more vulnerable to energy deficiency than SNc somas. During L-DOPA therapy, it was observed that higher L-DOPA dosage results in increased loss of terminals in SNc. It was also observed that co-administration of L-DOPA and glutathione (antioxidant) evades L-DOPA-induced toxicity in SNc neurons. Our proposed model of the SNc-striatum system is the first of its kind, where SNc neurons were modeled at a biophysical level, and striatal neurons were modeled at a spiking level. We show that our proposed model was able to capture L-DOPA-induced toxicity in SNc, caused by energy deficiency.

A dynamical model for the basal ganglia-thalamo-cortical oscillatory activity and its implications in Parkinson's disease

We propose to investigate brain electrophysiological alterations associated with Parkinson’s disease through a novel adaptive dynamical model of the network of the basal ganglia, the cortex and the thalamus. The model uniquely unifies the influence of dopamine in the regulation of the activity of all basal ganglia nuclei, the self-organised neuronal interdependent activity of basal ganglia-thalamo-cortical circuits and the generation of subcortical background oscillations. Variations in the amount of dopamine produced in the neurons of the substantia nigra pars compacta are key both in the onset of Parkinson’s disease and in the basal ganglia action selection. We model these dopamine-induced relationships, and Parkinsonian states are interpreted as spontaneous emergent behaviours associated with different rhythms of oscillatory activity patterns of the basal ganglia-thalamo-cortical network. These results are significant because: (1) the neural populations are built upon single-neuron models that have been robustly designed to have eletrophysiologically-realistic responses, and (2) our model distinctively links changes in the oscillatory activity in subcortical structures, dopamine levels in the basal ganglia and pathological synchronisation neuronal patterns compatible with Parkinsonian states, this still remains an open problem and is crucial to better understand the progression of the disease.

Ankle Push-Off Based Mathematical Model for Freezing of Gait in Parkinson's Disease

Freezing is an involuntary stopping of gait observed in late-stage Parkinson's disease (PD) patients. This is a highly debilitating symptom lacking a clear understanding of its causes. Walking in these patients is also associated with high variability, making both prediction of freezing and its understanding difficult. A neuromechanical model describes the motion of the mechanical (motor) aspects of the body under the action of neuromuscular forcing. In this work, a simplified neuromechanical model of gait is used to infer the causes for both the observed variability and freezing in PD. The mathematical model consists of the stance leg (during walking) modeled as a simple inverted pendulum acted upon by the ankle-push off forces from the trailing leg and pathological forces by the plantar-flexors of the stance leg. We model the effect on walking of the swing leg in the biped model and provide a rationale for using an inverted pendulum model. Freezing and irregular walking is demonstrated in the biped model as well as the inverted pendulum model. The inverted pendulum model is further studied semi-analytically to show the presence of horseshoe and chaos. While the plantar flexors of the swing leg push the center of mass (CoM) forward, the plantar flexors of the stance leg generate an opposing torque. Our study reveals that these opposing forces generated by the plantar flexors can induce freezing. Other gait abnormalities nearer to freezing such as a reduction in step length, and irregular walking patterns can also be explained by the model.

A Multi-Scale Computational Model of Excitotoxic Loss of Dopaminergic Cells in Parkinson's Disease

Parkinson's disease (PD) is a neurodegenerative disorder caused by loss of dopaminergic neurons in substantia nigra pars compacta (SNc). Although the exact cause of cell death is not clear, the hypothesis that metabolic deficiency is a key factor has been gaining attention in recent years. In the present study, we investigated this hypothesis using a multi-scale computational model of the subsystem of the basal ganglia comprising the subthalamic nucleus (STN), globus pallidus externa (GPe), and SNc. The proposed model is a multiscale model in that interaction among the three nuclei are simulated using more abstract Izhikevich neuron models, while the molecular pathways involved in cell death of SNc neurons are simulated in terms of detailed chemical kinetics. Simulation results obtained from the proposed model showed that energy deficiencies occurring at cellular and network levels could precipitate the excitotoxic loss of SNc neurons in PD. At the subcellular level, the models show how calcium elevation leads to apoptosis of SNc neurons. The therapeutic effects of several neuroprotective interventions are also simulated in the model. From neuroprotective studies, it was clear that glutamate inhibition and apoptotic signal blocker therapies were able to halt the progression of SNc cell loss when compared to other therapeutic interventions, which only slowed down the progression of SNc cell loss.

A biologically constrained spiking neural network model of the primate basal ganglia with overlapping pathways exhibits action selection

Action selection has been hypothesized to be a key function of the basal ganglia, yet the nuclei involved, their interactions and the importance of the direct/indirect pathway segregation in such process remain debated. Here, we design a spiking computational model of the monkey basal ganglia derived from a previously published population model, initially parameterized to reproduce electrophysiological activity at rest and to embody as much quantitative anatomical data as possible. As a particular feature, both models exhibit the strong overlap between the direct and indirect pathways that has been documented in non-human primates. Here, we first show how the translation from a population to an individual neuron model was achieved, with the addition of a minimal number of parameters. We then show that our model performs action selection, even though it was built without any assumption on the activity carried out during behaviour. We investigate the mechanisms of this selection through circuit disruptions and found an instrumental role of the off-centre/on-surround structure of the MSN-STN-GPi circuit, as well as of the MSN-MSN and FSI-MSN projections. This validates their potency in enabling selection. We finally study the pervasive centromedian and parafascicular thalamic inputs that reach all basal ganglia nuclei and whose influence is therefore difficult to anticipate. Our model predicts that these inputs modulate the responsiveness of action selection, making them a candidate for the regulation of the speed-accuracy trade-off during decision-making.

A review of computational modeling and deep brain stimulation: applications to Parkinson's disease

Biophysical computational models are complementary to experiments and theories, providing powerful tools for the study of neurological diseases. The focus of this review is the dynamic modeling and control strategies of Parkinson's disease (PD). In previous studies, the development of parkinsonian network dynamics modeling has made great progress. Modeling mainly focuses on the cortex-thalamus-basal ganglia (CTBG) circuit and its sub-circuits, which helps to explore the dynamic behavior of the parkinsonian network, such as synchronization. Deep brain stimulation (DBS) is an effective strategy for the treatment of PD. At present, many studies are based on the side effects of the DBS. However, the translation from modeling results to clinical disease mitigation therapy still faces huge challenges. Here, we introduce the progress of DBS improvement. Its specific purpose is to develop novel DBS treatment methods, optimize the treatment effect of DBS for each patient, and focus on the study in closed-loop DBS. Our goal is to review the inspiration and insights gained by combining the system theory with these computational models to analyze neurodynamics and optimize DBS treatment.

Deep Brain Stimulation of the Subthalamic Nucleus Induces Impulsive Responses to Bursts of Sensory Evidence

Decisions are made through the integration of external and internal inputs until a threshold is reached, triggering a response. The subthalamic nucleus (STN) has been implicated in adjusting the decision bound to prevent impulsivity during difficult decisions. We combine model-based and model-free approaches to test the theory that the STN raises the decision bound, a process impaired by deep brain stimulation (DBS). Eight male and female human subjects receiving treatment for Parkinson's disease with bilateral DBS of the STN performed an auditory two-alternative forced choice task. By ending trials unpredictably, we collected reaction time (RT) trials in which subjects reached their decision bound and non-RT trials in which subjects were forced to make a decision with less evidence. A decreased decision bound would cause worse performance on RT trials, and we found this to be the case on left-sided RT trials. Drift diffusion modeling showed a negative drift rate. This implies that in the absence of new evidence, the amount of evidence accumulated tends to drift toward zero. If evidence is accumulated at a constant rate this results in the evidence accumulated reaching an asymptote, the distance of which from the bound was decreased by DBS (p = 0.0079, random shuffle test), preventing subjects from controlling impulsivity. Subjects were more impulsive to bursts of stimuli associated with conflict (p < 0.001, cluster mass test). In addition, DBS lowered the decision bound specifically after error trials, decreasing the probability of switching to a non-RT trial after an error compared to correct response (28% vs. 38%, p = 0.005, Fisher exact test). The STN appears to function in decision-making by modulating the decision bound and drift rate to allow the suppression of impulsive responses.

A Computational Model of Loss of Dopaminergic Cells in Parkinson's Disease Due to Glutamate-Induced Excitotoxicity

Parkinson's disease (PD) is a neurodegenerative disease associated with progressive and inexorable loss of dopaminergic cells in Substantia Nigra pars compacta (SNc). Although many mechanisms have been suggested, a decisive root cause of this cell loss is unknown. A couple of the proposed mechanisms, however, show potential for the development of a novel line of PD therapeutics. One of these mechanisms is the peculiar metabolic vulnerability of SNc cells compared to other dopaminergic clusters; the other is the SubThalamic Nucleus (STN)-induced excitotoxicity in SNc. To investigate the latter hypothesis computationally, we developed a spiking neuron network-model of SNc-STN-GPe system. In the model, prolonged stimulation of SNc cells by an overactive STN leads to an increase in ‘stress' variable; when the stress in a SNc neuron exceeds a stress threshold, the neuron dies. The model shows that the interaction between SNc and STN involves a positive-feedback due to which, an initial loss of SNc cells that crosses a threshold causes a runaway-effect, leading to an inexorable loss of SNc cells, strongly resembling the process of neurodegeneration. The model further suggests a link between the two aforementioned mechanisms of SNc cell loss. Our simulation results show that the excitotoxic cause of SNc cell loss might initiate by weak-excitotoxicity mediated by energy deficit, followed by strong-excitotoxicity, mediated by a disinhibited STN. A variety of conventional therapies were simulated to test their efficacy in slowing down SNc cell loss. Among them, glutamate inhibition, dopamine restoration, subthalamotomy and deep brain stimulation showed superior neuroprotective-effects in the proposed model.

Roles for globus pallidus externa revealed in a computational model of action selection in the basal ganglia

The basal ganglia are considered vital to action selection - a hypothesis supported by several biologically plausible computational models. Of the several subnuclei of the basal ganglia, the globus pallidus externa (GPe) has been thought of largely as a relay nucleus, and its intrinsic connectivity has not been incorporated in significant detail, in any model thus far. Here, we incorporate newly revealed subgroups of neurons within the GPe into an existing computational model of the basal ganglia, and investigate their role in action selection. Three main results ensued. First, using previously used metrics for selection, the new extended connectivity improved the action selection performance of the model. Second, low frequency theta oscillations were observed in the subpopulation of the GPe (the TA or 'arkypallidal' neurons) which project exclusively to the striatum. These oscillations were suppressed by increased dopamine activity - revealing a possible link with symptoms of Parkinson's disease. Third, a new phenomenon was observed in which the usual monotonic relationship between input to the basal ganglia and its output within an action 'channel' was, under some circumstances, reversed. Thus, at high levels of input, further increase of this input to the channel could cause an increase of the corresponding output rather than the more usually observed decrease. Moreover, this phenomenon was associated with the prevention of multiple channel selection, thereby assisting in optimal action selection. Examination of the mechanistic origin of our results showed the so-called 'prototypical' GPe neurons to be the principal subpopulation influencing action selection. They control the striatum via the arkypallidal neurons and are also able to regulate the output nuclei directly. Taken together, our results highlight the role of the GPe as a major control hub of the basal ganglia, and provide a mechanistic account for its control function.

The role of cortical oscillations in a spiking neural network model of the basal ganglia

Although brain oscillations involving the basal ganglia (BG) have been the target of extensive research, the main focus lies disproportionally on oscillations generated within the BG circuit rather than other sources, such as cortical areas. We remedy this here by investigating the influence of various cortical frequency bands on the intrinsic effective connectivity of the BG, as well as the role of the latter in regulating cortical behaviour. To do this, we construct a detailed neural model of the complete BG circuit based on fine-tuned spiking neurons, with both electrical and chemical synapses as well as short-term plasticity between structures. As a measure of effective connectivity, we estimate information transfer between nuclei by means of transfer entropy. Our model successfully reproduces firing and oscillatory behaviour found in both the healthy and Parkinsonian BG. We found that, indeed, effective connectivity changes dramatically for different cortical frequency bands and phase offsets, which are able to modulate (or even block) information flow in the three major BG pathways. In particular, alpha (8–12Hz) and beta (13–30Hz) oscillations activate the direct BG pathway, and favour the modulation of the indirect and hyper-direct pathways via the subthalamic nucleus—globus pallidus loop. In contrast, gamma (30–90Hz) frequencies block the information flow from the cortex completely through activation of the indirect pathway. Finally, below alpha, all pathways decay gradually and the system gives rise to spontaneous activity generated in the globus pallidus. Our results indicate the existence of a multimodal gating mechanism at the level of the BG that can be entirely controlled by cortical oscillations, and provide evidence for the hypothesis of cortically-entrained but locally-generated subthalamic beta activity. These two findings suggest new insights into the pathophysiology of specific BG disorders.

Investigating Synchronous Oscillation and Deep Brain Stimulation Treatment in A Model of Cortico-Basal Ganglia Network

Altered firing properties and increased pathological oscillations in the basal ganglia have been proven to be hallmarks of Parkinson's disease (PD). Increasing evidence suggests that abnormal synchronous oscillations and suppression in the cortex may also play a critical role in the pathogenic process and treatment of PD. In this paper, a new closed-loop network including the cortex and basal ganglia using the Izhikevich models is proposed to investigate the synchrony and pathological oscillations in motor circuits and their modulation by deep brain stimulation (DBS). Results show that more coherent dynamics in the cortex may cause stronger effects on the synchrony and pathological oscillations of the subthalamic nucleus (STN). The pathological beta oscillations of the STN can both be efficiently suppressed with DBS applied directly to the STN or to cortical neurons, respectively, but the underlying mechanisms by which DBS suppresses the beta oscillations are different. This research helps to understand the dynamics of pathological oscillations in PD-related motor regions and supports the therapeutic potential of stimulation of cortical neurons.

Reward Based Motor Adaptation Mediated by Basal Ganglia

It is widely accepted that the basal ganglia (BG) play a key role in action selection and reinforcement learning. However, despite considerable number of studies, the BG architecture and function are not completely understood. Action selection and reinforcement learning are facilitated by the activity of dopaminergic neurons, which encode reward prediction errors when reward outcomes are higher or lower than expected. The BG are thought to select proper motor responses by gating appropriate actions, and suppressing inappropriate ones. The direct striato-nigral (GO) and the indirect striato-pallidal (NOGO) pathways have been suggested to provide the functions of BG in the two-pathway concept. Previous models confirmed the idea that these two pathways can mediate the behavioral choice, but only for a relatively small number of potential behaviors. Recent studies have provided new evidence of BG involvement in motor adaptation tasks, in which adaptation occurs in a non-error-based manner. In such tasks, there is a continuum of possible actions, each represented by a complex neuronal activity pattern. We extended the classical concept of the two-pathway BG by creating a model of BG interacting with a movement execution system, which allows for an arbitrary number of possible actions. The model includes sensory and premotor cortices, BG, a spinal cord network, and a virtual mechanical arm performing 2D reaching movements. The arm is composed of 2 joints (shoulder and elbow) controlled by 6 muscles (4 mono-articular and 2 bi-articular). The spinal cord network contains motoneurons, controlling the muscles, and sensory interneurons that receive afferent feedback and mediate basic reflexes. Given a specific goal-oriented motor task, the BG network through reinforcement learning constructs a behavior from an arbitrary number of basic actions represented by cortical activity patterns. Our study confirms that, with slight modifications, the classical two-pathway BG concept is consistent with results of previous studies, including non-error based motor adaptation experiments, pharmacological manipulations with BG nuclei, and functional deficits observed in BG-related motor disorders.

Decision making under uncertainty in a spiking neural network model of the basal ganglia

The mechanisms of decision-making and action selection are generally thought to be under the control of parallel cortico-subcortical loops connecting back to distinct areas of cortex through the basal ganglia and processing motor, cognitive and limbic modalities of decision-making. We have used these properties to develop and extend a connectionist model at a spiking neuron level based on a previous rate model approach. This model is demonstrated on decision-making tasks that have been studied in primates and the electrophysiology interpreted to show that the decision is made in two steps. To model this, we have used two parallel loops, each of which performs decision-making based on interactions between positive and negative feedback pathways. This model is able to perform two-level decision-making as in primates. We show here that, before learning, synaptic noise is sufficient to drive the decision-making process and that, after learning, the decision is based on the choice that has proven most likely to be rewarded. The model is then submitted to lesion tests, reversal learning and extinction protocols. We show that, under these conditions, it behaves in a consistent manner and provides predictions in accordance with observed experimental data.

Electrode Position and Current Amplitude Modulate Impulsivity after Subthalamic Stimulation in Parkinsons Disease-A Computational Study

Background: Subthalamic Nucleus Deep Brain Stimulation (STN-DBS) is highly effective in alleviating motor symptoms of Parkinson's disease (PD) which are not optimally controlled by dopamine replacement therapy. Clinical studies and reports suggest that STN-DBS may result in increased impulsivity and de novo impulse control disorders (ICD).

Objective/Hypothesis: We aimed to compare performance on a decision making task, the Iowa Gambling Task (IGT), in healthy conditions (HC), untreated and medically-treated PD conditions with and without STN stimulation. We hypothesized that the position of electrode and stimulation current modulate impulsivity after STN-DBS.

Methods: We built a computational spiking network model of basal ganglia (BG) and compared the model's STN output with STN activity in PD. Reinforcement learning methodology was applied to simulate IGT performance under various conditions of dopaminergic and STN stimulation where IGT total and bin scores were compared among various conditions.

Results: The computational model reproduced neural activity observed in normal and PD conditions. Untreated and medically-treated PD conditions had lower total IGT scores (higher impulsivity) compared to HC (P < 0.0001). The electrode position that happens to selectively stimulate the part of the STN corresponding to an advantageous panel on IGT resulted in de-selection of that panel and worsening of performance (P < 0.0001). Supratherapeutic stimulation amplitudes also worsened IGT performance (P < 0.001).

Conclusion(s): In our computational model, STN stimulation led to impulsive decision making in IGT in PD condition. Electrode position and stimulation current influenced impulsivity which may explain the variable effects of STN-DBS reported in patients.

Forgetting in Reinforcement Learning Links Sustained Dopamine Signals to Motivation

It has been suggested that dopamine (DA) represents reward-prediction-error (RPE) defined in reinforcement learning and therefore DA responds to unpredicted but not predicted reward. However, recent studies have found DA response sustained towards predictable reward in tasks involving self-paced behavior, and suggested that this response represents a motivational signal. We have previously shown that RPE can sustain if there is decay/forgetting of learned-values, which can be implemented as decay of synaptic strengths storing learned-values. This account, however, did not explain the suggested link between tonic/sustained DA and motivation. In the present work, we explored the motivational effects of the value-decay in self-paced approach behavior, modeled as a series of ‘Go’ or ‘No-Go’ selections towards a goal. Through simulations, we found that the value-decay can enhance motivation, specifically, facilitate fast goal-reaching, albeit counterintuitively. Mathematical analyses revealed that underlying potential mechanisms are twofold: (1) decay-induced sustained RPE creates a gradient of ‘Go’ values towards a goal, and (2) value-contrasts between ‘Go’ and ‘No-Go’ are generated because while chosen values are continually updated, unchosen values simply decay. Our model provides potential explanations for the key experimental findings that suggest DA's roles in motivation: (i) slowdown of behavior by post-training blockade of DA signaling, (ii) observations that DA blockade severely impairs effortful actions to obtain rewards while largely sparing seeking of easily obtainable rewards, and (iii) relationships between the reward amount, the level of motivation reflected in the speed of behavior, and the average level of DA. These results indicate that reinforcement learning with value-decay, or forgetting, provides a parsimonious mechanistic account for the DA's roles in value-learning and motivation. Our results also suggest that when biological systems for value-learning are active even though learning has apparently converged, the systems might be in a state of dynamic equilibrium, where learning and forgetting are balanced.

Dopamine (DA) has been suggested to have two reward-related roles: (1) representing reward-prediction-error (RPE), and (2) providing motivational drive. Role(1) is based on the physiological results that DA responds to unpredicted but not predicted reward, whereas role(2) is supported by the pharmacological results that blockade of DA signaling causes motivational impairments such as slowdown of self-paced behavior. So far, these two roles are considered to be played by two different temporal patterns of DA signals: role(1) by phasic signals and role(2) by tonic/sustained signals. However, recent studies have found sustained DA signals with features indicative of both roles (1) and (2), complicating this picture. Meanwhile, whereas synaptic/circuit mechanisms for role(1), i.e., how RPE is calculated in the upstream of DA neurons and how RPE-dependent update of learned-values occurs through DA-dependent synaptic plasticity, have now become clarified, mechanisms for role(2) remain unclear. In this work, we modeled self-paced behavior by a series of ‘Go’ or ‘No-Go’ selections in the framework of reinforcement-learning assuming DA's role(1), and demonstrated that incorporation of decay/forgetting of learned-values, which is presumably implemented as decay of synaptic strengths storing learned-values, provides a potential unified mechanistic account for the DA's two roles, together with its various temporal patterns.

Probing the Role of Medication, DBS Electrode Position, and Antidromic Activation on Impulsivity Using a Computational Model of Basal Ganglia

Everyday, we encounter situations where available choices are nearly equally rewarding (high conflict) calling for some tough decision making. Experimental recordings showed that the activity of Sub Thalamic Nucleus (STN) increases during such situations providing the extra time needed to make the right decision, teasing apart the most rewarding choice from the runner up closely trailing behind. This prolonged deliberation necessary for decision making under high conflict was absent in Parkinson's disease (PD) patients who underwent Deep Brain Stimulation (DBS) surgery of STN. In an attempt to understand the underlying cause of such adverse response, we built a 2D spiking network model (50 × 50 lattice) of Basal ganglia incorporating the key nuclei. Using the model we studied the Probabilistic learning task (PLT) in untreated, treated (L-Dopa and Dopamine Agonist) and STN-DBS PD conditions. Based on the experimental observation that dopaminergic activity is analogous to temporal difference (TD) and induces cortico-striatal plasticity, we introduced learning in the cortico-striatal weights. The results show that healthy and untreated conditions of PD model were able to more or less equally select (avoid) the rewarding (punitive) choice, a behavior that was absent in treated PD condition. The time taken to select a choice in high conflict trials was high in normal condition, which is in agreement with experimental results. The treated PD (Dopamine Agonist) patients made impulsive decisions (small reaction time) which in turn led to poor performance. The underlying cause of the observed impulsivity in DBS patients was studied in the model by (1) varying the electrode position within STN, (2) causing antidromic activation of GPe neurons. The effect of electrode position on reaction time was analyzed by studying the activity of STN neurons where, a decrease in STN neural activity was observed for certain electrode positions. We also observed that a higher antidromic activation of GPe neurons does not impact the learning ability but decreases reaction time as reported in DBS patients. These results suggest a probable role of electrode and antidromic activation in modulating the STN activity and eventually affecting the patient's performance on PLT.

Basal Ganglia dysfunctions in movement disorders: What can be learned from computational simulations

The basal ganglia are a complex neuronal system that is impaired in several movement disorders, including Parkinson's disease, Huntington's disease, and dystonia. Empirical studies have provided valuable insights into the brain dysfunctions underlying these disorders. The systems-level perspective, however, of how patients' motor, cognitive, and emotional impairments originate from known brain dysfunctions has been a challenge to empirical investigations. These causal relations have been analyzed via computational modeling, a method that describes the simulation of interacting brain processes in a computer system. In this article, we review computational insights into the brain dysfunctions underlying Parkinson's disease, Huntington's disease, and dystonia, with particular foci on dysfunctions of the dopamine system, basal ganglia pathways, and neuronal oscillations. © 2016 International Parkinson and Movement Disorder Society.