Interactions of glutamate and dopamine in a computational model of the striatum

A silent eligibility trace enables dopamine-dependent synaptic plasticity for reinforcement learning in the mouse striatum

Dopamine‐dependent synaptic plasticity is a candidate mechanism for reinforcement learning. A silent eligibility trace – initiated by synaptic activity and transformed into synaptic strengthening by later action of dopamine – has been hypothesized to explain the retroactive effect of dopamine in reinforcing past behaviour. We tested this hypothesis by measuring time‐dependent modulation of synaptic plasticity by dopamine in adult mouse striatum, using whole‐cell recordings. Presynaptic activity followed by postsynaptic action potentials (pre–post) caused spike‐timing‐dependent long‐term depression in D1‐expressing neurons, but not in D2 neurons, and not if postsynaptic activity followed presynaptic activity. Subsequent experiments focused on D1 neurons. Applying a dopamine D1 receptor agonist during induction of pre–post plasticity caused long‐term potentiation. This long‐term potentiation was hidden by long‐term depression occurring concurrently and was unmasked when long‐term depression blocked an L‐type calcium channel antagonist. Long‐term potentiation was blocked by a Ca²⁺‐permeable AMPA receptor antagonist but not by an NMDA antagonist or an L‐type calcium channel antagonist. Pre–post stimulation caused transient elevation of rectification – a marker for expression of Ca²⁺‐permeable AMPA receptors – for 2–4‐s after stimulation. To test for an eligibility trace, dopamine was uncaged at specific time points before and after pre‐ and postsynaptic conjunction of activity. Dopamine caused potentiation selectively at synapses that were active 2‐s before dopamine release, but not at earlier or later times. Our results provide direct evidence for a silent eligibility trace in the synapses of striatal neurons. This dopamine‐timing‐dependent plasticity may play a central role in reinforcement learning.

Solving the Credit Assignment Problem With the Prefrontal Cortex

In naturalistic multi-cue and multi-step learning tasks, where outcomes of behavior are delayed in time, discovering which choices are responsible for rewards can present a challenge, known as the credit assignment problem. In this review, I summarize recent work that highlighted a critical role for the prefrontal cortex (PFC) in assigning credit where it is due in tasks where only a few of the multitude of cues or choices are relevant to the final outcome of behavior. Collectively, these investigations have provided compelling support for specialized roles of the orbitofrontal (OFC), anterior cingulate (ACC), and dorsolateral prefrontal (dlPFC) cortices in contingent learning. However, recent work has similarly revealed shared contributions and emphasized rich and heterogeneous response properties of neurons in these brain regions. Such functional overlap is not surprising given the complexity of reciprocal projections spanning the PFC. In the concluding section, I overview the evidence suggesting that the OFC, ACC and dlPFC communicate extensively, sharing the information about presented options, executed decisions and received rewards, which enables them to assign credit for outcomes to choices on which they are contingent. This account suggests that lesion or inactivation/inhibition experiments targeting a localized PFC subregion will be insufficient to gain a fine-grained understanding of credit assignment during learning and instead poses refined questions for future research, shifting the focus from focal manipulations to experimental techniques targeting cortico-cortical projections.

Significance of input correlations in striatal function

The striatum is the main input station of the basal ganglia and is strongly associated with motor and cognitive functions. Anatomical evidence suggests that individual striatal neurons are unlikely to share their inputs from the cortex. Using a biologically realistic large-scale network model of striatum and cortico-striatal projections, we provide a functional interpretation of the special anatomical structure of these projections. Specifically, we show that weak pairwise correlation within the pool of inputs to individual striatal neurons enhances the saliency of signal representation in the striatum. By contrast, correlations among the input pools of different striatal neurons render the signal representation less distinct from background activity. We suggest that for the network architecture of the striatum, there is a preferred cortico-striatal input configuration for optimal signal representation. It is further enhanced by the low-rate asynchronous background activity in striatum, supported by the balance between feedforward and feedback inhibitions in the striatal network. Thus, an appropriate combination of rates and correlations in the striatal input sets the stage for action selection presumably implemented in the basal ganglia.

Conditional routing of information to the cortex: a model of the basal ganglia's role in cognitive coordination

The basal ganglia play a central role in cognition and are involved in such general functions as action selection and reinforcement learning. Here, we present a model exploring the hypothesis that the basal ganglia implement a conditional information-routing system. The system directs the transmission of cortical signals between pairs of regions by manipulating separately the selection of sources and destinations of information transfers. We suggest that such a mechanism provides an account for several cognitive functions of the basal ganglia. The model also incorporates a possible mechanism by which subsequent transfers of information control the release of dopamine. This signal is used to produce novel stimulus-response associations by internalizing transferred cortical representations in the striatum. We discuss how the model is related to production systems and cognitive architectures. A series of simulations is presented to illustrate how the model can perform simple stimulus-response tasks, develop automatic behaviors, and provide an account of impairments in Parkinson's and Huntington's diseases.

Computational models of reinforcement learning: the role of dopamine as a reward signal

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.

A kinetic model of dopamine- and calcium-dependent striatal synaptic plasticity

Corticostriatal synapse plasticity of medium spiny neurons is regulated by glutamate input from the cortex and dopamine input from the substantia nigra. While cortical stimulation alone results in long-term depression (LTD), the combination with dopamine switches LTD to long-term potentiation (LTP), which is known as dopamine-dependent plasticity. LTP is also induced by cortical stimulation in magnesium-free solution, which leads to massive calcium influx through NMDA-type receptors and is regarded as calcium-dependent plasticity. Signaling cascades in the corticostriatal spines are currently under investigation. However, because of the existence of multiple excitatory and inhibitory pathways with loops, the mechanisms regulating the two types of plasticity remain poorly understood. A signaling pathway model of spines that express D1-type dopamine receptors was constructed to analyze the dynamic mechanisms of dopamine- and calcium-dependent plasticity. The model incorporated all major signaling molecules, including dopamine- and cyclic AMP-regulated phosphoprotein with a molecular weight of 32 kDa (DARPP32), as well as AMPA receptor trafficking in the post-synaptic membrane. Simulations with dopamine and calcium inputs reproduced dopamine- and calcium-dependent plasticity. Further in silico experiments revealed that the positive feedback loop consisted of protein kinase A (PKA), protein phosphatase 2A (PP2A), and the phosphorylation site at threonine 75 of DARPP-32 (Thr75) served as the major switch for inducing LTD and LTP. Calcium input modulated this loop through the PP2B (phosphatase 2B)-CK1 (casein kinase 1)-Cdk5 (cyclin-dependent kinase 5)-Thr75 pathway and PP2A, whereas calcium and dopamine input activated the loop via PKA activation by cyclic AMP (cAMP). The positive feedback loop displayed robust bi-stable responses following changes in the reaction parameters. Increased basal dopamine levels disrupted this dopamine-dependent plasticity. The present model elucidated the mechanisms involved in bidirectional regulation of corticostriatal synapses and will allow for further exploration into causes and therapies for dysfunctions such as drug addiction.

Recent brain imaging and neurophysiological studies suggest that the striatum, the start of the basal ganglia circuit, plays a major role in value-based decision making and behavioral disorders such as drug addiction. The plasticity of synaptic input from the cerebral cortex to output neurons of the striatum, which are medium spiny neurons, depends on interactions between glutamate input from the cortex and dopaminergic input from the midbrain. It also links sensory and cognitive states in the cortex with reward-oriented action outputs. The mechanisms involved in molecular cascades that transmit glutamate and dopamine inputs to changes in postsynaptic glutamate receptors are very complex and it is difficult to intuitively understand the mechanism. Therefore, a biochemical network model was constructed, and computer simulations were performed. The model reproduced dopamine-dependent and calcium-dependent forms of long-term depression (LTD) and potentiation (LTP) of corticostriatal synapses. Further in silico experiments revealed that a positive feedback loop formed by proteins, the protein specifically expressed in the striatum, served as the major switch for inducing LTD and LTP. This model could allow us to understand dynamic constraints in reward-dependent learning, as well as causes and therapies of dopamine-related disorders such as drug addiction.

Dopamine receptor activation is required for corticostriatal spike-timing-dependent plasticity

Single action potentials (APs) backpropagate into the higher-order dendrites of striatal spiny projection neurons during cortically driven "up" states. The timing of these backpropagating APs relative to the arriving corticostriatal excitatory inputs determines changes in dendritic calcium concentration. The question arises to whether this spike-timing relative to cortical excitatory inputs can also induce synaptic plasticity at corticostriatal synapses. Here we show that timing of single postsynaptic APs relative to the cortically evoked EPSP determines both the direction and the strength of synaptic plasticity in spiny projection neurons. Single APs occurring 30 ms before the cortically evoked EPSP induced long-term depression (LTD), whereas APs occurring 10 ms after the EPSP induced long-term potentiation (LTP). The amount of plasticity decreased as the time between the APs and EPSPs was increased, with the resulting spike-timing window being broader for LTD than for LTP. In addition, we show that dopamine receptor activation is required for this spike-timing-dependent plasticity (STDP). Blocking dopamine D(1)/D(5) receptors prevented both LTD and LTP induction. In contrast, blocking dopamine D(2) receptors delayed, but did not prevent, LTD and sped induction of LTP. We conclude (1) that, in combination with cortical inputs, single APs evoked in spiny projection neurons can induce both LTP and LTD of the corticostriatal pathway; (2) that the strength and direction of these synaptic changes depend deterministically on the AP timing relative to the arriving cortical inputs; (3) that, whereas dopamine D(2) receptor activation modulates the initial phase of striatal STDP, dopamine D(1)/D(5) receptor activation is critically required for striatal STDP. Thus, the timing of APs relative to cortical inputs alone is not enough to induce corticostriatal plasticity, implying that ongoing activity does not affect synaptic strength unless dopamine receptors are activated.

Simulation of GABA function in the basal ganglia: computational models of GABAergic mechanisms in basal ganglia function

This chapter outlines current interpretation of computational aspects of GABAergic circuits of the striatum. Recent hypotheses and controversial matters are reviewed. Quantitative aspects of striatal synaptology relevant to computational models are considered, with estimates of the connectivity of the spiny projection neurons and fast-spiking interneurons. Against this background, insights into the computational properties of inhibitory circuits based on analysis and simulation of simple models are discussed. The paper concludes with suggestions for further theoretical and experimental studies.

DARPP-32 is a robust integrator of dopamine and glutamate signals

Integration of neurotransmitter and neuromodulator signals in the striatum plays a central role in the functions and dysfunctions of the basal ganglia. DARPP-32 is a key actor of this integration in the GABAergic medium-size spiny neurons, in particular in response to dopamine and glutamate. When phosphorylated by cAMP-dependent protein kinase (PKA), DARPP-32 inhibits protein phosphatase-1 (PP1), whereas when phosphorylated by cyclin-dependent kinase 5 (CDK5) it inhibits PKA. DARPP-32 is also regulated by casein kinases and by several protein phosphatases. These complex and intricate regulations make simple predictions of DARPP-32 dynamic behaviour virtually impossible. We used detailed quantitative modelling of the regulation of DARPP-32 phosphorylation to improve our understanding of its function. The models included all the combinations of the three best-characterized phosphorylation sites of DARPP-32, their regulation by kinases and phosphatases, and the regulation of those enzymes by cAMP and Ca(2+) signals. Dynamic simulations allowed us to observe the temporal relationships between cAMP and Ca(2+) signals. We confirmed that the proposed regulation of protein phosphatase-2A (PP2A) by calcium can account for the observed decrease of Threonine 75 phosphorylation upon glutamate receptor activation. DARPP-32 is not simply a switch between PP1-inhibiting and PKA-inhibiting states. Sensitivity analysis showed that CDK5 activity is a major regulator of the response, as previously suggested. Conversely, the strength of the regulation of PP2A by PKA or by calcium had little effect on the PP1-inhibiting function of DARPP-32 in these conditions. The simulations showed that DARPP-32 is not only a robust signal integrator, but that its response also depends on the delay between cAMP and calcium signals affecting the response to the latter. This integration did not depend on the concentration of DARPP-32, while the absolute effect on PP1 varied linearly. In silico mutants showed that Ser137 phosphorylation affects the influence of the delay between dopamine and glutamate, and that constitutive phosphorylation in Ser137 transforms DARPP-32 in a quasi-irreversible switch. This work is a first attempt to better understand the complex interactions between cAMP and Ca(2+) regulation of DARPP-32. Progressive inclusion of additional components should lead to a realistic model of signalling networks underlying the function of striatal neurons.

Computational models of the basal ganglia: from robots to membranes

With the rapid accumulation of neuroscientific data comes a pressing need to develop models that can explain the computational processes performed by the basal ganglia. Relevant biological information spans a range of structural levels, from the activity of neuronal membranes to the role of the basal ganglia in overt behavioural control. This viewpoint presents a framework for understanding the aims, limitations and methods for testing of computational models across all structural levels. We identify distinct modelling strategies that can deliver important and complementary insights into the nature of problems the basal ganglia have evolved to solve, and describe methods that are used to solve them.

Neuroengineering models of brain disease

The techniques of computational simulation have begun to be applied to modeling neurological disease and mental illness. Such neuroengineering models provide a conceptual bridge between molecular/cellular pathology and cognitive performance. We consider models of Alzheimer's disease, Parkinson's disease, and schizophrenia. Each of these diseases involves a disorder of neuromodulation coupled with underlying neuronal pathology. Parallels arising between these models suggests that a common set of computational mechanisms may account for functional loss across a spectrum of brain diseases. In particular, we focus on attractor-based network dynamics and how they arise from neural architectures, on mechanisms for linking sequences of attractor states and their role in cognition, and on the role of neuromodulation in controlling these processes. These studies suggest new approaches to understanding the forebrain circuits underlying cognition, and point toward a new tool for dissecting the pathophysiology of brain disease.

A possible functional organization of the corticostriatal input within the weakly-correlated striatal activity: a modeling study

Recently, it was reported in an in vivo study that pairs of the striatal projection neurons (medium-sized spiny neurons) of the basal ganglia show asynchronous spiking within weakly-correlated subthreshold depolarized states. In this computational study, we investigate a possible functional organization of corticostriatal inputs that accounts for the experimental observations within known anatomical and physiological constraints. In a pair of medium-sized spiny neurons, a small fraction of corticostriatal fibers is common to both neurons. To explain the weak correlations in sub- and supra-threshold activities of the neuron pair, we postulate that the two input channels, common or specific to the individual neurons, have distinct functional roles. The common input channel delivers random spike trains and is primarily responsible for the initiation and maintenance of the depolarized states. In contrast, the input through the neuron-specific channels elicit postsynaptic spikes by delivering intermittently-synchronized spikes. The results of this model were compared with those derived from a newly-performed analysis of the previous double-intracellular recording data. We show that the behavior of this model agrees qualitatively and quantitatively with that of the medium-sized spiny neurons observed in the experiments in a certain range of the coincident time window.

Striatal mechanism of action of corticoliberin on behavior in dogs in conditions of dopamine deficiency

This report describes studies of the interaction of the integrative dopaminergic and corticoliberin systems in the neostriatum during performance of situational food-related conditioned reflexes. Studies were performed in dogs with chemotrodes implanted in the substantia nigra and the head of the caudate nucleus. 6-Hydroxydopamine was injected into the substantia nigra at a dose of 50 microg, and 10 microg of corticoliberin was injected into the caudate nucleus. Blood cortisol and catecholamine levels were determined. Analysis of the result showed that an interaction takes place in the neostriatum between the corticoliberin and dopaminergic systems, and that in conditions in which dopaminergic structures are excluded, the efficacy of corticoliberin in the performance of behavioral acts decreases by 30-40%, i.e., complete expression of its regulatory role of motor situational conditioned reflexes is lost.

A predictive reinforcement model of dopamine neurons for learning approach behavior

A neural network model of how dopamine and prefrontal cortex activity guides short- and long-term information processing within the cortico-striatal circuits during reward-related learning of approach behavior is proposed. The model predicts two types of reward-related neuronal responses generated during learning: (1) cell activity signaling errors in the prediction of the expected time of reward delivery and (2) neural activations coding for errors in the prediction of the amount and type of reward or stimulus expectancies. The former type of signal is consistent with the responses of dopaminergic neurons, while the latter signal is consistent with reward expectancy responses reported in the prefrontal cortex. It is shown that a neural network architecture that satisfies the design principles of the adaptive resonance theory of Carpenter and Grossberg (1987) can account for the dopamine responses to novelty, generalization, and discrimination of appetitive and aversive stimuli. These hypotheses are scrutinized via simulations of the model in relation to the delivery of free food outside a task, the timed contingent delivery of appetitive and aversive stimuli, and an asymmetric, instructed delay response task.

A database for cell signaling networks

We developed a data and knowledge base for cellular signal transduction in human cells, to make this rapidly growing information available. The database includes all the biological properties of cellular signal transduction, including biological reactions that transfer cellular signals and molecular attributes characterized by sequences, structures, and functions. Since the database is based on the object-oriented technique, highly flexible methods of data definition and modification are necessary to handle this diverse and complex biological information. The database includes attractive graphical representations of signaling cascades and the three-dimensional structure of molecules. The database is a novel application of ACEDB, which was the database originally developed to store the C. elegans genome. The database can be accessed through the Internet at http://geo.nihs.go.jp/csndb.html.

Striatal mechanisms in Parkinson's disease: new insights from computer modeling

We review data and hypotheses concerning the functional anatomy of the striatum and the role of its corticostriatal and nigrostriatal afferents in Parkinson's disease (PD). Starting from molecular mechanisms of glutamatergic and dopaminergic actions in the striatum we have developed a compartmental model of striatal principal neurons that displays a significant degree of biological realism. Simulations of a network of striatal projection neurons under conditions likely to be found in healthy subjects as well as untreated and therapeutic situations of advanced PD provide clues concerning the dynamics of neuronal interactions and their possible effects on downstream motor structures in the generation of positive and negative motor symptoms. We present tentative biological explanations of the symptoms of rigidity and akinesia in PD leading to predictions concerning the origin of abnormal movements and the beneficial effects of dopaminergic treatment. Although these attempts are not yet sufficient to account for the complexity of clinical symptoms found in PD they can guide further empirical research and foster fruitful interactions between experimentalists, theoreticians, and clinicians in unraveling the functional anatomy of the basal ganglia.

Up and down states in striatal medium spiny neurons simultaneously recorded with spontaneous activity in fast-spiking interneurons studied in cortex-striatum-substantia nigra organotypic cultures

In vivo intracellular spontaneous activity in striatal medium spiny (MS) projection neurons is characterized by "up" and "down" states. How this type of activity relates to the neuronal activity of striatal fast-spiking (FS) interneurons was examined in the presence of nigral and cortical inputs using cortex-striatum-substantia nigra organotypic cultures grown for 45 +/- 4 d. The nigrostriatal projection was confirmed by tyrosine hydroxylase immunoreactivity. Corticostriatal (CS) projection neurons, striatal MS neurons, and FS neurons were intracellularly recorded and morphologically and electrophysiologically characterized. Intracellular spontaneous activity in the cultures consisted of intermittent depolarized periods of 0.5-1 sec duration. Spontaneous depolarizations in MS neurons were restricted to a narrow membrane potential range (up state) during which they occasionally fired single spikes. These up states were completely blocked by the glutamate antagonist CNQX. In FS interneurons, depolarized periods were characterized by large membrane potential fluctuations that occupied a wide range between rest and spike threshold. Also, FS interneurons spontaneously fired at much higher rates than did MS neurons. Simultaneous intracellular recordings established that during spontaneous depolarizations MS neurons and FS interneurons displayed correlated subthreshold neuronal activity in the low frequency range. These results indicate that (1) the CS projection neurons, striatal MS neurons, and FS interneurons grown in cortex-striatum-substantia nigra organotypic cultures show morphological and electrophysiological characteristics similar to those seen in vivo; (2) striatal MS neurons but not FS interneurons show an up state; (3) striatal MS neurons and FS interneurons receive common, presumably cortical inputs in the low frequency range. Our results support the view that the cortex provides a feedforward inhibition of MS neuron activity during the up state via FS interneurons.

Network models of the basal ganglia

Over the past year, a number of conceptual and mathematical models of the basal ganglia and their interactions with other areas of the brain have appeared in the literature. Even though the models each differ in significant ways, several computational principles, such as convergence, recurrence and competition, appear to have emerged as common themes of information processing in the basal ganglia. Simulation studies of these models have provoked new types of questions at the many levels of inquiry linking biophysics to behavior.