A silent eligibility trace enables dopamine-dependent synaptic plasticity for reinforcement learning in the mouse striatum

Astrocyte D1/D5 Dopamine Receptors Govern Non-Hebbian Long-Term Potentiation at Sensory Synapses onto Lamina I Spinoparabrachial Neurons

Recent work demonstrated that activation of spinal D1 and D5 dopamine receptors (D1/D5Rs) facilitates non-Hebbian long-term potentiation (LTP) at primary afferent synapses onto spinal projection neurons. However, the cellular localization of the D1/D5Rs driving non-Hebbian LTP in spinal nociceptive circuits remains unknown, and it is also unclear whether D1/D5R signaling must occur concurrently with sensory input in order to promote non-Hebbian LTP at these synapses. Here we investigate these issues using cell-type-selective knockdown of D1Rs or D5Rs from lamina I spinoparabrachial neurons, dorsal root ganglion (DRG) neurons, or astrocytes in adult mice of either sex using Cre recombinase-based genetic strategies. The LTP evoked by low-frequency stimulation of primary afferents in the presence of the selective D1/D5R agonist SKF82958 persisted following the knockdown of D1R or D5R in spinoparabrachial neurons, suggesting that postsynaptic D1/D5R signaling was dispensable for non-Hebbian plasticity at sensory synapses onto these key output neurons of the superficial dorsal horn (SDH). Similarly, the knockdown of D1Rs or D5Rs in DRG neurons failed to influence SKF82958-enabled LTP in lamina I projection neurons. In contrast, SKF82958-induced LTP was suppressed by the knockdown of D1R or D5R in spinal astrocytes. Furthermore, the data indicate that the activation of D1R/D5Rs in spinal astrocytes can either retroactively or proactively drive non-Hebbian LTP in spinoparabrachial neurons. Collectively, these results suggest that dopaminergic signaling in astrocytes can strongly promote activity-dependent LTP in the SDH, which is predicted to significantly enhance the amplification of ascending nociceptive transmission from the spinal cord to the brain.

Synchronous Measurements of Extracellular Action Potentials and Neurochemical Activity with Carbon Fiber Electrodes in Nonhuman Primates

Measuring the dynamic relationship between neuromodulators, such as dopamine, and neuronal action potentials is imperative to understand how these fundamental modes of neural signaling interact to mediate behavior. We developed methods to measure concurrently dopamine and extracellular action potentials (i.e., spikes) in monkeys. Standard fast-scan cyclic voltammetric (FSCV) electrochemical (EChem) and electrophysiological (EPhys) recording systems are combined and used to collect spike and dopamine signals, respectively, from an array of carbon fiber (CF) sensors implanted in the monkey striatum. FSCV requires the application of small voltages at the implanted sensors to measure redox currents generated from target molecules, such as dopamine. These applied voltages create artifacts at neighboring EPhys measurement sensors which may lead to misclassification of these signals as physiological spikes. Therefore, simple automated temporal interpolation algorithms were designed to remove these artifacts and enable accurate spike extraction. We validated these methods using simulated artifacts and demonstrated an average spike recovery rate of 84.5%. We identified and discriminated cell type-specific units in the monkey striatum that were shown to correlate to specific behavioral task parameters related to reward size and eye movement direction. Synchronously recorded spike and dopamine signals displayed contrasting relations to the task variables, suggesting a complex relationship between these two modes of neural signaling. Future application of our methods will help advance our understanding of the interactions between neuromodulator signaling and neuronal activity, to elucidate more detailed mechanisms of neural circuitry and plasticity mediating behaviors in health and in disease.

Distinct dopaminergic spike-timing-dependent plasticity rules are suited to different functional roles

Various mathematical models have been formulated to describe the changes in synaptic strengths resulting from spike-timing-dependent plasticity (STDP). A subset of these models include a third factor, dopamine, which interacts with the timing of pre- and postsynaptic spiking to contribute to plasticity at specific synapses, notably those from cortex to striatum at the input layer of the basal ganglia. Theoretical work to analyze these plasticity models has largely focused on abstract issues, such as the conditions under which they may promote synchronization and the weight distributions induced by inputs with simple correlation structures, rather than on scenarios associated with specific tasks, and has generally not considered dopamine-dependent forms of STDP. In this paper, we analyze, mathematically and with simulations, three forms of dopamine-modulated STDP in three scenarios that are relevant to corticostriatal synapses. Two of the models considered comprise previously proposed STDP rules with modifications to incorporate dopamine, while the third is a corticostriatal dopamine-dependent STDP rule adapted from a similar one already in the literature. We test the ability of each of the three models to maintain its weights in the face of noise and to complete simple reward prediction and action selection tasks, studying the learned weight distributions and corresponding task performance in each setting. Interestingly, we find that each of the three plasticity rules is well suited to a subset of the scenarios studied but falls short in others. These results show that different tasks may require different forms of synaptic plasticity, yielding the prediction that the precise form of the STDP mechanism may vary across regions of the striatum, and other brain areas impacted by dopamine, that are involved in distinct computational functions.

What Role Does Striatal Dopamine Play in Goal-directed Action?

Evidence suggests that dopamine activity provides a US-related prediction error for Pavlovian conditioning and the reinforcement signal supporting the acquisition of habits. However, its role in goal-directed action is less clear. There are currently few studies that have assessed dopamine release as animals acquire and perform self-paced instrumental actions. Here we briefly review the literature documenting the psychological, behavioral and neural bases of goal-directed actions in rats and mice, before turning to describe recent studies investigating the role of dopamine in instrumental learning and performance. Plasticity in dorsomedial striatum, a central node in the network supporting goal-directed action, clearly requires dopamine release, the timing of which, relative to cortical and thalamic inputs, determines the degree and form of that plasticity. Beyond this, bilateral release appears to reflect reward prediction errors as animals experience the consequences of an action. Such signals feedforward to update the value of the specific action associated with that outcome during subsequent performance, with dopamine release at the time of action reflecting the updated predicted action value. More recently, evidence has also emerged for a hemispherically lateralised signal associated with the action; dopamine release is greater in the hemisphere contralateral to the spatial target of the action. This effect emerges over the course of acquisition and appears to reflect the strength of the action-outcome association. Thus, during goal-directed action, dopamine release signals the action, the outcome and their association to shape the learning and performance processes necessary to support this form of behavioral control.

Dynamic behaviour restructuring mediates dopamine-dependent credit assignment

Animals exhibit a diverse behavioural repertoire when exploring new environments and can learn which actions or action sequences produce positive outcomes. Dopamine release after encountering a reward is critical for reinforcing reward-producing actions1-3. However, it has been challenging to understand how credit is assigned to the exact action that produced the dopamine release during continuous behaviour. Here we investigated this problem in mice using a self-stimulation paradigm in which specific spontaneous movements triggered optogenetic stimulation of dopaminergic neurons. Dopamine self-stimulation rapidly and dynamically changes the structure of the entire behavioural repertoire. Initial stimulations reinforced not only the stimulation-producing target action, but also actions similar to the target action and actions that occurred a few seconds before stimulation. Repeated pairings led to a gradual refinement of the behavioural repertoire to home in on the target action. Reinforcement of action sequences revealed further temporal dependencies of refinement. Action pairs spontaneously separated by long time intervals promoted a stepwise credit assignment, with early refinement of actions most proximal to stimulation and subsequent refinement of more distal actions. Thus, a retrospective reinforcement mechanism promotes not only reinforcement, but also gradual refinement of the entire behavioural repertoire to assign credit to specific actions and action sequences that lead to dopamine release.

Using rodent data to elucidate dopaminergic mechanisms of ADHD: Implications for human personality

An altered behavioral response to positive reinforcement has been proposed to be a core deficit in attention deficit hyperactivity disorder (ADHD). The spontaneously hypertensive rat (SHR), a congenic animal strain, displays a similarly altered response to reinforcement. The presence of this genetically determined phenotype in a rodent model allows experimental investigation of underlying neural mechanisms. Behaviorally, the SHR displays increased preference for immediate reinforcement, increased sensitivity to individual instances of reinforcement relative to integrated reinforcement history, and a steeper delay of reinforcement gradient compared to other rat strains. The SHR also shows less development of incentive to approach sensory stimuli, or cues, that predict reward after repeated cue-reward pairing. We consider the underlying neural mechanisms for these characteristics. It is well known that midbrain dopamine neurons are initially activated by unexpected reward and gradually transfer their responses to reward-predicting cues. This finding has inspired the dopamine transfer deficit (DTD) hypothesis, which predicts certain behavioral effects that would arise from a deficient transfer of dopamine responses from actual rewards to reward-predicting cues. We argue that the DTD predicts the altered responses to reinforcement seen in the SHR and individuals with ADHD. These altered responses to reinforcement in turn predict core symptoms of ADHD. We also suggest that variations in the degree of dopamine transfer may underlie variations in personality dimensions related to altered reinforcement sensitivity. In doing so, we highlight the value of rodent models to the study of human personality.

Sensory Reinforced Corticostriatal Plasticity

BACKGROUND

Regional changes in corticostriatal transmission induced by phasic dopaminergic signals are an essential feature of the neural network responsible for instrumental reinforcement during discovery of an action. However, the timing of signals that are thought to contribute to the induction of corticostriatal plasticity is difficult to reconcile within the framework of behavioural reinforcement learning, because the reinforcer is normally delayed relative to the selection and execution of causally-related actions.

OBJECTIVE

While recent studies have started to address the relevance of delayed reinforcement signals and their impact on corticostriatal processing, our objective was to establish a model in which a sensory reinforcer triggers appropriately delayed reinforcement signals relayed to the striatum via intact neuronal pathways and to investigate the effects on corticostriatal plasticity.

METHODS

We measured corticostriatal plasticity with electrophysiological recordings using a light flash as a natural sensory reinforcer, and pharmacological manipulations were applied in an in vivo anesthetized rat model preparation.

RESULTS

We demonstrate that the spiking of striatal neurons evoked by single-pulse stimulation of the motor cortex can be potentiated by a natural sensory reinforcer, operating through intact afferent pathways, with signal timing approximating that required for behavioural reinforcement. The pharmacological blockade of dopamine receptors attenuated the observed potentiation of corticostriatal neurotransmission.

CONCLUSION

This novel in vivo model of corticostriatal plasticity offers a behaviourally relevant framework to address the physiological, anatomical, cellular, and molecular bases of instrumental reinforcement learning.

Synchronous Measurements of Extracellular Action Potentials and Neurochemical Activity with Carbon Fiber Electrodes in Nonhuman Primates

Measuring the dynamic relationship between neuromodulators, such as dopamine, and neuronal action potentials is imperative to understand how these fundamental modes of neural signaling interact to mediate behavior. Here, we developed methods to measure concurrently dopamine and extracellular action potentials (i.e., spikes) and applied these in a monkey performing a behavioral task. Standard fast-scan cyclic voltammetric (FSCV) electrochemical (EChem) and electrophysiological (EPhys) recording systems are combined and used to collect spike and dopamine signals, respectively, from an array of carbon fiber (CF) sensors implanted in the monkey striatum. FSCV requires the application of small voltages at the implanted sensors to measure redox currents generated from target molecules, such as dopamine. These applied voltages create artifacts at neighboring EPhys-measurement sensors, producing signals that may falsely be classified as physiological spikes. Therefore, simple automated temporal interpolation algorithms were designed to remove these artifacts and enable accurate spike extraction. We validated these methods using simulated artifacts and demonstrated an average spike recovery rate of 84.5%. This spike extraction was performed on data collected from concurrent EChem and EPhys recordings made in a task-performing monkey to discriminate cell-type specific striatal units. These identified units were shown to correlate to specific behavioral task parameters related to reward size and eye-movement direction. Synchronous measures of spike and dopamine signals displayed contrasting relations to the behavioral task parameters, as taken from our small set of representative data, suggesting a complex relationship between these two modes of neural signaling. Future application of our methods will help advance our understanding of the interactions between neuromodulator signaling and neuronal activity, to elucidate more detailed mechanisms of neural circuitry and plasticity mediating behaviors in health and in disease.

A critical role for CaMKII in behavioral timescale synaptic plasticity in hippocampal CA1 pyramidal neurons

Behavioral timescale synaptic plasticity (BTSP) is a type of non-Hebbian synaptic plasticity reported to underlie place field formation. Despite this important function, the molecular mechanisms underlying BTSP are poorly understood. The α-calcium-calmodulin-dependent protein kinase II (αCaMKII) is activated by synaptic transmission-mediated calcium influx, and its subsequent phosphorylation is central to synaptic plasticity. Because the activity of αCaMKII is known to outlast the event triggering phosphorylation, we hypothesized that it could mediate the extended timescale of BTSP. To examine the role of αCaMKII in BTSP, we performed whole-cell in vivo and in vitro recordings in CA1 pyramidal neurons from mice engineered with a point mutation at the autophosphorylation site (T286A) causing accelerated signaling kinetics. Here, we demonstrate a profound deficit in synaptic plasticity, strongly suggesting that αCaMKII signaling is required for BTSP. This study elucidates part of the molecular mechanism of BTSP and provides insight into the function of αCaMKII in place cell formation and ultimately learning and memory.

The Dysfunctional Mechanisms Throwing Tics: Structural and Functional Changes in Tourette Syndrome

Tourette Syndrome (TS) is a high-incidence multifactorial neuropsychiatric disorder characterized by motor and vocal tics co-occurring with several diverse comorbidities, including obsessive-compulsive disorder and attention-deficit hyperactivity disorder. The origin of TS is multifactorial, with strong genetic, perinatal, and immunological influences. Although almost all neurotransmettitorial systems have been implicated in TS pathophysiology, a comprehensive neurophysiological model explaining the dynamics of expression and inhibition of tics is still lacking. The genesis and maintenance of motor and non-motor aspects of TS are thought to arise from functional and/or structural modifications of the basal ganglia and related circuitry. This complex wiring involves several cortical and subcortical structures whose concerted activity controls the selection of the most appropriate reflexive and habitual motor, cognitive and emotional actions. Importantly, striatal circuits exhibit bidirectional forms of synaptic plasticity that differ in many respects from hippocampal and neocortical plasticity, including sensitivity to metaplastic molecules such as dopamine. Here, we review the available evidence about structural and functional anomalies in neural circuits which have been found in TS patients. Finally, considering what is known in the field of striatal plasticity, we discuss the role of exuberant plasticity in TS, including the prospect of future pharmacological and neuromodulation avenues.

The mesoanatomy of the cortex, minimization of free energy, and generative cognition

Capacity for generativity and unlimited association is the defining characteristic of sentience, and this capacity somehow arises from neuronal self-organization in the cortex. We have previously argued that, consistent with the free energy principle, cortical development is driven by synaptic and cellular selection maximizing synchrony, with effects manifesting in a wide range of features of mesoscopic cortical anatomy. Here, we further argue that in the postnatal stage, as more structured inputs reach the cortex, the same principles of self-organization continue to operate at multitudes of local cortical sites. The unitary ultra-small world structures that emerged antenatally can represent sequences of spatiotemporal images. Local shifts of presynapses from excitatory to inhibitory cells result in the local coupling of spatial eigenmodes and the development of Markov blankets, minimizing prediction errors in each unit's interactions with surrounding neurons. In response to the superposition of inputs exchanged between cortical areas, more complicated, potentially cognitive structures are competitively selected by the merging of units and the elimination of redundant connections that result from the minimization of variational free energy and the elimination of redundant degrees of freedom. The trajectory along which free energy is minimized is shaped by interaction with sensorimotor, limbic, and brainstem mechanisms, providing a basis for creative and unlimited associative learning.

The contribution of the basal ganglia and cerebellum to motor learning: A neuro-computational approach

Motor learning involves a widespread brain network including the basal ganglia, cerebellum, motor cortex, and brainstem. Despite its importance, little is known about how this network learns motor tasks and which role different parts of this network take. We designed a systems-level computational model of motor learning, including a cortex-basal ganglia motor loop and the cerebellum that both determine the response of central pattern generators in the brainstem. First, we demonstrate its ability to learn arm movements toward different motor goals. Second, we test the model in a motor adaptation task with cognitive control, where the model replicates human data. We conclude that the cortex-basal ganglia loop learns via a novelty-based motor prediction error to determine concrete actions given a desired outcome, and that the cerebellum minimizes the remaining aiming error.

Cellular bases for reward-related dopamine actions

Dopamine neurons exhibit transient increases and decreases in their firing rate upon reward and punishment for learning. This bidirectional modulation of dopamine dynamics occurs on the order of hundreds of milliseconds, and it is sensitively detected to reinforce the preceding sensorimotor events. These observations indicate that the mechanisms of dopamine detection at the projection sites are of remarkable precision, both in time and concentration. A major target of dopamine projection is the striatum, including the ventral region of the nucleus accumbens, which mainly comprises dopamine D1 and D2 receptor (D1R and D2R)-expressing spiny projection neurons. Although the involvement of D1R and D2R in dopamine-dependent learning has been suggested, the exact cellular bases for detecting transient dopamine signaling remain unclear. This review discusses recent cellular studies on the novel synaptic mechanisms for detecting dopamine transient signals associated with learning. Analyses of behavior based on these mechanisms have further revealed new behavioral aspects that are closely associated with these synaptic mechanisms. Thus, it is gradually possible to mechanistically explain behavioral learning via synaptic and cellular bases in rodents.

Glutamatergic neurotransmission: A potential pharmacotherapeutic target for the treatment of cognitive disorders

In the mammalian brain, glutamate is regarded to be the primary excitatory neurotransmitter due to its widespread distribution and wide range of metabolic functions. Glutamate plays key roles in regulating neurogenesis, synaptogenesis, neurite outgrowth, and neuron survival in the brain. Ionotropic and metabotropic glutamate receptors, neurotransmitters, neurotensin, neurosteroids, and others co-ordinately formulate a complex glutamatergic network in the brain that maintains optimal excitatory neurotransmission. Cognitive activities are potentially synchronized by the glutamatergic activities in the brain via restoring synaptic plasticity. Dysfunctional glutamate receptors and other glutamatergic components are responsible for the aberrant glutamatergic activity in the brain that cause cognitive impairments, loss of synaptic plasticity, and neuronal damage. Thus, controlling the brain's glutamatergic transmission and modifying glutamate receptor function could be a potential therapeutic strategy for cognitive disorders. Certain drugs that regulate glutamate receptor activities have shown therapeutic promise in improving cognitive functions in preclinical and clinical studies. However, several issues regarding precise functional information of glutamatergic activity are yet to be comprehensively understood. The present article discusses the scope of developing glutamatergic systems as prospective pharmacotherapeutic targets to treat cognitive disorders. Special attention has been given to recent developments, challenges, and future prospects.

Mesolimbic dopamine adapts the rate of learning from action

Recent success in training artificial agents and robots derives from a combination of direct learning of behavioural policies and indirect learning through value functions^1-3. Policy learning and value learning use distinct algorithms that optimize behavioural performance and reward prediction, respectively. In animals, behavioural learning and the role of mesolimbic dopamine signalling have been extensively evaluated with respect to reward prediction⁴; however, so far there has been little consideration of how direct policy learning might inform our understanding⁵. Here we used a comprehensive dataset of orofacial and body movements to understand how behavioural policies evolved as naive, head-restrained mice learned a trace conditioning paradigm. Individual differences in initial dopaminergic reward responses correlated with the emergence of learned behavioural policy, but not the emergence of putative value encoding for a predictive cue. Likewise, physiologically calibrated manipulations of mesolimbic dopamine produced several effects inconsistent with value learning but predicted by a neural-network-based model that used dopamine signals to set an adaptive rate, not an error signal, for behavioural policy learning. This work provides strong evidence that phasic dopamine activity can regulate direct learning of behavioural policies, expanding the explanatory power of reinforcement learning models for animal learning⁶.

Adaptive control of synaptic plasticity integrates micro- and macroscopic network function

Synaptic plasticity configures interactions between neurons and is therefore likely to be a primary driver of behavioral learning and development. How this microscopic-macroscopic interaction occurs is poorly understood, as researchers frequently examine models within particular ranges of abstraction and scale. Computational neuroscience and machine learning models offer theoretically powerful analyses of plasticity in neural networks, but results are often siloed and only coarsely linked to biology. In this review, we examine connections between these areas, asking how network computations change as a function of diverse features of plasticity and vice versa. We review how plasticity can be controlled at synapses by calcium dynamics and neuromodulatory signals, the manifestation of these changes in networks, and their impacts in specialized circuits. We conclude that metaplasticity-defined broadly as the adaptive control of plasticity-forges connections across scales by governing what groups of synapses can and can't learn about, when, and to what ends. The metaplasticity we discuss acts by co-opting Hebbian mechanisms, shifting network properties, and routing activity within and across brain systems. Asking how these operations can go awry should also be useful for understanding pathology, which we address in the context of autism, schizophrenia and Parkinson's disease.

Continuous cholinergic-dopaminergic updating in the nucleus accumbens underlies approaches to reward-predicting cues

The ability to learn Pavlovian associations from environmental cues predicting positive outcomes is critical for survival, motivating adaptive behaviours. This cued-motivated behaviour depends on the nucleus accumbens (NAc). NAc output activity mediated by spiny projecting neurons (SPNs) is regulated by dopamine, but also by cholinergic interneurons (CINs), which can release acetylcholine and glutamate via the activity of the vesicular acetylcholine transporter (VAChT) or the vesicular glutamate transporter (VGLUT3), respectively. Here we investigated behavioural and neurochemical changes in mice performing a touchscreen Pavlovian approach task by recording dopamine, acetylcholine, and calcium dynamics from D1- and D2-SPNs using fibre photometry in control, VAChT or VGLUT3 mutant mice to understand how these signals cooperate in the service of approach behaviours toward reward-predicting cues. We reveal that NAc acetylcholine-dopaminergic signalling is continuously updated to regulate striatal output underlying the acquisition of Pavlovian approach learning toward reward-predicting cues.

The positive valence system, adaptive behaviour and the origins of reward

Although the hey-day of motivation as an area of study is long past, the issues with which motivational theorists grappled have not grown less important: i.e. the development of deterministic explanations for the particular tuning of the nervous system to specific changes in the internal and external environment and the organisation of adaptive behavioural responses to those changes. Here, we briefly elaborate these issues in describing the structure and function of the 'positive valence system'. We describe the origins of adaptive behaviour in an ascending arousal system, sensitive to peripheral regulatory changes, that modulates and activates various central motivational states. Associations between these motivational states and sensory inputs underlie evaluative conditioning and generate the representation of the 'unconditioned' stimuli fundamental to Pavlovian conditioning. As a consequence, associations with these stimuli can generate Pavlovian conditioned responses through the motivational control of stimulus event associations with sensory and affective components of the valence system to elicit conditioned orienting, consummatory and preparatory responses, particularly the affective responses reflecting Pavlovian excitation and inhibition, arousal and reinforcement, the latter used to control the formation of habits. These affective processes also provoke emotional responses, allowing the externalisation of positive valence in hedonic experience to generate the goal or reward values that mediate goal-directed action. Together these processes form the positive valence system, ensure the maintenance of adaptive behaviour and, through the association of sensory events and emotional responses through consummatory experience, provide the origins of reward.

Dendritic involvement in inhibition and disinhibition of vulnerable dopaminergic neurons in healthy and pathological conditions

Dopaminergic neurons in the substantia nigra pars compacta (SNc) differentially degenerate in Parkinson's Disease, with the ventral region degenerating more severely than the dorsal region. Compared with the dorsal neurons, the ventral neurons in the SNc have distinct dendritic morphology, electrophysiological characteristics, and circuit connections with the basal ganglia. These characteristics shape information processing in the ventral SNc and structure the balance of inhibition and disinhibition in the striatonigral circuitry. In this paper, I review foundational studies and recent work comparing the circuitry of the ventral and dorsal SNc neurons and discuss how loss of the ventral neurons early in Parkinson's Disease could affect the overall balance of inhibition and disinhibition of dopamine signals.

Synaptic Plasticity Is Predicted by Spatiotemporal Firing Rate Patterns and Robust to In Vivo-like Variability

Synaptic plasticity, the experience-induced change in connections between neurons, underlies learning and memory in the brain. Most of our understanding of synaptic plasticity derives from in vitro experiments with precisely repeated stimulus patterns; however, neurons exhibit significant variability in vivo during repeated experiences. Further, the spatial pattern of synaptic inputs to the dendritic tree influences synaptic plasticity, yet is not considered in most synaptic plasticity rules. Here, we investigate how spatiotemporal synaptic input patterns produce plasticity with in vivo-like conditions using a data-driven computational model with a plasticity rule based on calcium dynamics. Using in vivo spike train recordings as inputs to different size clusters of spines, we show that plasticity is strongly robust to trial-to-trial variability of spike timing. In addition, we derive general synaptic plasticity rules describing how spatiotemporal patterns of synaptic inputs control the magnitude and direction of plasticity. Synapses that strongly potentiated have greater firing rates and calcium concentration later in the trial, whereas strongly depressing synapses have hiring firing rates early in the trial. The neighboring synaptic activity influences the direction and magnitude of synaptic plasticity, with small clusters of spines producing the greatest increase in synaptic strength. Together, our results reveal that calcium dynamics can unify diverse plasticity rules and reveal how spatiotemporal firing rate patterns control synaptic plasticity.

Fiber photometry in striatum reflects primarily nonsomatic changes in calcium

Fiber photometry enables recording of population neuronal calcium dynamics in awake mice. While the popularity of fiber photometry has grown in recent years, it remains unclear whether photometry reflects changes in action potential firing (that is, 'spiking') or other changes in neuronal calcium. In microscope-based calcium imaging, optical and analytical approaches can help differentiate somatic from neuropil calcium. However, these approaches cannot be readily applied to fiber photometry. As such, it remains unclear whether the fiber photometry signal reflects changes in somatic calcium, changes in nonsomatic calcium or a combination of the two. Here, using simultaneous in vivo extracellular electrophysiology and fiber photometry, along with in vivo endoscopic one-photon and two-photon calcium imaging, we determined that the striatal fiber photometry does not reflect spiking-related changes in calcium and instead primarily reflects nonsomatic changes in calcium.

Ramping activity in the striatum

Control of the timing of behavior is thought to require the basal ganglia (BG) and BG pathologies impair performance in timing tasks. Temporal interval discrimination depends on the ramping activity of medium spiny neurons (MSN) in the main BG input structure, the striatum, but the underlying mechanisms driving this activity are unclear. Here, we combine an MSN dynamical network model with an action selection system applied to an interval discrimination task. We find that when network parameters are appropriate for the striatum so that slowly fluctuating marginally stable dynamics are intrinsically generated, up and down ramping populations naturally emerge which enable significantly above chance task performance. We show that emergent population activity is in very good agreement with empirical studies and discuss how MSN network dysfunction in disease may alter temporal perception.

Norepinephrine potentiates and serotonin depresses visual cortical responses by transforming eligibility traces

Reinforcement allows organisms to learn which stimuli predict subsequent biological relevance. Hebbian mechanisms of synaptic plasticity are insufficient to account for reinforced learning because neuromodulators signaling biological relevance are delayed with respect to the neural activity associated with the stimulus. A theoretical solution is the concept of eligibility traces (eTraces), silent synaptic processes elicited by activity which upon arrival of a neuromodulator are converted into a lasting change in synaptic strength. Previously we demonstrated in visual cortical slices the Hebbian induction of eTraces and their conversion into LTP and LTD by the retroactive action of norepinephrine and serotonin Here we show in vivo in mouse V1 that the induction of eTraces and their conversion to LTP/D by norepinephrine and serotonin respectively potentiates and depresses visual responses. We also show that the integrity of this process is crucial for ocular dominance plasticity, a canonical model of experience-dependent plasticity.

Training a spiking neuronal network model of visual-motor cortex to play a virtual racket-ball game using reinforcement learning

Recent models of spiking neuronal networks have been trained to perform behaviors in static environments using a variety of learning rules, with varying degrees of biological realism. Most of these models have not been tested in dynamic visual environments where models must make predictions on future states and adjust their behavior accordingly. The models using these learning rules are often treated as black boxes, with little analysis on circuit architectures and learning mechanisms supporting optimal performance. Here we developed visual/motor spiking neuronal network models and trained them to play a virtual racket-ball game using several reinforcement learning algorithms inspired by the dopaminergic reward system. We systematically investigated how different architectures and circuit-motifs (feed-forward, recurrent, feedback) contributed to learning and performance. We also developed a new biologically-inspired learning rule that significantly enhanced performance, while reducing training time. Our models included visual areas encoding game inputs and relaying the information to motor areas, which used this information to learn to move the racket to hit the ball. Neurons in the early visual area relayed information encoding object location and motion direction across the network. Neuronal association areas encoded spatial relationships between objects in the visual scene. Motor populations received inputs from visual and association areas representing the dorsal pathway. Two populations of motor neurons generated commands to move the racket up or down. Model-generated actions updated the environment and triggered reward or punishment signals that adjusted synaptic weights so that the models could learn which actions led to reward. Here we demonstrate that our biologically-plausible learning rules were effective in training spiking neuronal network models to solve problems in dynamic environments. We used our models to dissect the circuit architectures and learning rules most effective for learning. Our model shows that learning mechanisms involving different neural circuits produce similar performance in sensory-motor tasks. In biological networks, all learning mechanisms may complement one another, accelerating the learning capabilities of animals. Furthermore, this also highlights the resilience and redundancy in biological systems.

Serotonin receptors contribute to dopamine depression of lateral inhibition in the nucleus accumbens

Dopamine modulation of nucleus accumbens (NAc) circuitry is central to theories of reward seeking and reinforcement learning. Despite decades of effort, the acute dopamine actions on the NAc microcircuitry remain puzzling. Here, we dissect out the direct actions of dopamine on lateral inhibition between medium spiny neurons (MSNs) in mouse brain slices and find that they are pathway specific. Dopamine potently depresses GABAergic transmission from presynaptic dopamine D2 receptor-expressing MSNs (D2-MSNs), whereas it potentiates transmission from presynaptic dopamine D1 receptor-expressing MSNs (D1-MSNs) onto other D1-MSNs. To our surprise, presynaptic D2 receptors mediate only half of the depression induced by endogenous and exogenous dopamine. Presynaptic serotonin 5-HT1B receptors are responsible for a significant component of dopamine-induced synaptic depression. This study clarifies the mechanistic understanding of dopamine actions in the NAc by showing pathway-specific modulation of lateral inhibition and involvement of D2 and 5-HT1B receptors in dopamine depression of D2-MSN synapses.

Animal models of action control and cognitive dysfunction in Parkinson's disease

Parkinson's disease (PD) has historically been considered a motor disorder induced by a loss of dopaminergic neurons in the substantia nigra pars compacta. More recently, it has been recognized to have significant non-motor symptoms, most prominently cognitive symptoms associated with a dysexecutive syndrome. It is common in the literature to see motor and cognitive symptoms treated separately and, indeed, there has been a general call for specialized treatment of the latter, particularly in the more severe cases of PD with mild cognitive impairment and dementia. Animal studies have similarly been developed to model the motor or non-motor symptoms. Nevertheless, considerable research has established that segregating consideration of cognition from the precursors to motor movement, particularly movement associated with goal-directed action, is difficult if not impossible. Indeed, on some contemporary views cognition is embodied in action control, something that is particularly prevalent in theory and evidence relating to the integration of goal-directed and habitual control processes. The current paper addresses these issues within the literature detailing animal models of cognitive dysfunction in PD and their neural and neurochemical bases. Generally, studies using animal models of PD provide some of the clearest evidence for the integration of these action control processes at multiple levels of analysis and imply that consideration of this integrative process may have significant benefits for developing new approaches to the treatment of PD.

The role of plasticity in the recovery of consciousness

Disorders of consciousness (DOCs), i.e., coma, vegetative state, and minimally conscious state are the consequences of a severe brain injury that disrupts the brain ability to generate consciousness. Recovery from DOCs requires functional and structural changes in the brain. The sites where these plastic changes take place vary according to the pathophysiology of the DOC. The ascending reticular activating system of the brainstem and its complex connections with the thalamus and cortex are involved in the pathophysiology of coma. Subcortical structures, such as the striatum and globus pallidus, together with thalamocortical and corticothalamic projections, the basal forebrain, and several networks among different cortical areas are probably involved in vegetative and minimally conscious states. Some mechanisms of plasticity that allegedly operate in each of these sites to promote recovery of consciousness will be discussed in this chapter. While some mechanisms of plasticity work at a local level, others produce functional changes in complex neuronal networks, for example by entraining neuronal oscillations. The specific mechanisms of brain plasticity represent potential targets for future treatments aiming to restore consciousness in patients with severe DOCs.

Frequency-Dependent Action of Neuromodulation

In oscillatory circuits, some actions of neuromodulators depend on the oscillation frequency. However, the mechanisms are poorly understood. We explored this problem by characterizing neuromodulation of the lateral pyloric (LP) neuron of the crab stomatogastric ganglion (STG). Many peptide modulators, including proctolin, activate the same ionic current (I_MI) in STG neurons. Because I_MI is fast and non-inactivating, its peak level does not depend on the temporal properties of neuronal activity. We found, however, that the amplitude and peak time of the proctolin-activated current in LP is frequency dependent. Because frequency affects the rate of voltage change, we measured these currents with voltage ramps of different slopes and found that proctolin activated two kinetically distinct ionic currents: the known I_MI, whose amplitude is independent of ramp slope or direction, and an inactivating current (I_MI-T), which was only activated by positive ramps and whose amplitude increased with increasing ramp slope. Using a conductance-based model we found that I_MI and I_MI-T make distinct contributions to the bursting activity, with I_MI increasing the excitability, and I_MI-T regulating the burst onset by modifying the postinhibitory rebound in a frequency-dependent manner. The voltage dependence and partial calcium permeability of I_MI-T is similar to other characterized neuromodulator-activated currents in this system, suggesting that these are isoforms of the same channel. Our computational model suggests that calcium permeability may allow this current to also activate the large calcium-dependent potassium current in LP, providing an additional mechanism to regulate burst termination. These results demonstrate a mechanism for frequency-dependent actions of neuromodulators.

Calcium-Permeable AMPA Receptors Mediate Timing-Dependent LTP Elicited by Low Repeat Coincident Pre- and Postsynaptic Activity at Schaffer Collateral-CA1 Synapses

High-frequency stimulation induced long-term potentiation (LTP) and low-frequency stimulation induced LTD are considered as cellular models of memory formation. Interestingly, spike timing-dependent plasticity (STDP) can induce equally robust timing-dependent LTP (t-LTP) and t-LTD in response to low frequency repeats of coincident action potential (AP) firing in presynaptic and postsynaptic cells. Commonly, STDP paradigms relying on 25-100 repeats of coincident AP firing are used to elicit t-LTP or t-LTD, but the minimum number of repeats required for successful STDP is barely explored. However, systematic investigation of physiologically relevant low repeat STDP paradigms is of utmost importance to explain learning mechanisms in vivo. Here, we examined low repeat STDP at Schaffer collateral-CA1 synapses by pairing one presynaptic AP with either one postsynaptic AP (1:1 t-LTP), or a burst of 4 APs (1:4 t-LTP) and found 3-6 repeats to be sufficient to elicit t-LTP. 6× 1:1 t-LTP required postsynaptic Ca2+ influx via NMDARs and L-type VGCCs and was mediated by increased presynaptic glutamate release. In contrast, 1:4 t-LTP depended on postsynaptic metabotropic GluRs and ryanodine receptor signaling and was mediated by postsynaptic insertion of AMPA receptors. Unexpectedly, both 6× t-LTP variants were strictly dependent on activation of postsynaptic Ca2+-permeable AMPARs but were differentially regulated by dopamine receptor signaling. Our data show that synaptic changes induced by only 3-6 repeats of mild STDP stimulation occurring in ≤10 s can take place on time scales observed also during single trial learning.

The critical balance between dopamine D2 receptor and RGS for the sensitive detection of a transient decay in dopamine signal

In behavioral learning, reward-related events are encoded into phasic dopamine (DA) signals in the brain. In particular, unexpected reward omission leads to a phasic decrease in DA (DA dip) in the striatum, which triggers long-term potentiation (LTP) in DA D2 receptor (D2R)-expressing spiny-projection neurons (D2 SPNs). While this LTP is required for reward discrimination, it is unclear how such a short DA-dip signal (0.5-2 s) is transferred through intracellular signaling to the coincidence detector, adenylate cyclase (AC). In the present study, we built a computational model of D2 signaling to determine conditions for the DA-dip detection. The DA dip can be detected only if the basal DA signal sufficiently inhibits AC, and the DA-dip signal sufficiently disinhibits AC. We found that those two requirements were simultaneously satisfied only if two key molecules, D2R and regulators of G protein signaling (RGS) were balanced within a certain range; this balance has indeed been observed in experimental studies. We also found that high level of RGS was required for the detection of a 0.5-s short DA dip, and the analytical solutions for these requirements confirmed their universality. The imbalance between D2R and RGS is associated with schizophrenia and DYT1 dystonia, both of which are accompanied by abnormal striatal LTP. Our simulations suggest that D2 SPNs in patients with schizophrenia and DYT1 dystonia cannot detect short DA dips. We finally discussed that such psychiatric and movement disorders can be understood in terms of the imbalance between D2R and RGS.

A Multisubcellular Compartment Model of AMPA Receptor Trafficking for Neuromodulation of Hebbian Synaptic Plasticity

Neuromodulation can profoundly impact the gain and polarity of postsynaptic changes in Hebbian synaptic plasticity. An emerging pattern observed in multiple central synapses is a pull–push type of control in which activation of receptors coupled to the G-protein Gs promote long-term potentiation (LTP) at the expense of long-term depression (LTD), whereas receptors coupled to Gq promote LTD at the expense of LTP. Notably, coactivation of both Gs- and Gq-coupled receptors enhances the gain of both LTP and LTD. To account for these observations, we propose a simple kinetic model in which AMPA receptors (AMPARs) are trafficked between multiple subcompartments in and around the postsynaptic spine. In the model AMPARs in the postsynaptic density compartment (PSD) are the primary contributors to synaptic conductance. During LTP induction, AMPARs are trafficked to the PSD primarily from a relatively small perisynaptic (peri-PSD) compartment. Gs-coupled receptors promote LTP by replenishing peri-PSD through increased AMPAR exocytosis from a pool of endocytic AMPAR. During LTD induction AMPARs are trafficked in the reverse direction, from the PSD to the peri-PSD compartment, and Gq-coupled receptors promote LTD by clearing the peri-PSD compartment through increased AMPAR endocytosis. We claim that the model not only captures essential features of the pull–push neuromodulation of synaptic plasticity, but it is also consistent with other actions of neuromodulators observed in slice experiments and is compatible with the current understanding of AMPAR trafficking.

Wave-like dopamine dynamics as a mechanism for spatiotemporal credit assignment

Significant evidence supports the view that dopamine shapes learning by encoding reward prediction errors. However, it is unknown whether striatal targets receive tailored dopamine dynamics based on regional functional specialization. Here, we report wave-like spatiotemporal activity patterns in dopamine axons and release across the dorsal striatum. These waves switch between activational motifs and organize dopamine transients into localized clusters within functionally related striatal subregions. Notably, wave trajectories were tailored to task demands, propagating from dorsomedial to dorsolateral striatum when rewards are contingent on animal behavior and in the opponent direction when rewards are independent of behavioral responses. We propose a computational architecture in which striatal dopamine waves are sculpted by inference about agency and provide a mechanism to direct credit assignment to specialized striatal subregions. Supporting model predictions, dorsomedial dopamine activity during reward-pursuit signaled the extent of instrumental control and interacted with reward waves to predict future behavioral adjustments.

Behavioral Time Scale Plasticity of Place Fields: Mathematical Analysis

Traditional synaptic plasticity experiments and models depend on tight temporal correlations between pre- and postsynaptic activity. These tight temporal correlations, on the order of tens of milliseconds, are incompatible with significantly longer behavioral time scales, and as such might not be able to account for plasticity induced by behavior. Indeed, recent findings in hippocampus suggest that rapid, bidirectional synaptic plasticity which modifies place fields in CA1 operates at behavioral time scales. These experimental results suggest that presynaptic activity generates synaptic eligibility traces both for potentiation and depression, which last on the order of seconds. These traces can be converted to changes in synaptic efficacies by the activation of an instructive signal that depends on naturally occurring or experimentally induced plateau potentials. We have developed a simple mathematical model that is consistent with these observations. This model can be fully analyzed to find the fixed points of induced place fields and how these fixed points depend on system parameters such as the size and shape of presynaptic place fields, the animal's velocity during induction, and the parameters of the plasticity rule. We also make predictions about the convergence time to these fixed points, both for induced and pre-existing place fields.

Environment-based object values learned by local network in the striatum tail

Choosing good objects is a fundamental behavior for all animals, to which the basal ganglia (BG) contribute extensively. However, the object choice needs to be changed in different environments. The mechanism of object choice is based on the neuronal circuits originating from output neurons (MSNs) in the striatum. We found that the environment information is provided by fast-spiking interneurons (FSIs) connecting to the MSN circuit. More critically, the experimental reduction of the FSI-input to MSNs disabled the monkey to learn the environment-based object choice. This proved that the object choice controlled by the downstream BG circuit is modulated by the environmental context controlled by the internal circuits in the top of BG circuit. This is important for our flexible decision.

Basal ganglia contribute to object-value learning, which is critical for survival. The underlying neuronal mechanism is the association of each object with its rewarding outcome. However, object values may change in different environments and we then need to choose different objects accordingly. The mechanism of this environment-based value learning is unknown. To address this question, we created an environment-based value task in which the value of each object was reversed depending on the two scene-environments (X and Y). After experiencing this task repeatedly, the monkeys became able to switch the choice of object when the scene-environment changed unexpectedly. When we blocked the inhibitory input from fast-spiking interneurons (FSIs) to medium spiny projection neurons (MSNs) in the striatum tail by locally injecting IEM-1460, the monkeys became unable to learn scene-selective object values. We then studied the mechanism of the FSI-MSN connection. Before and during this learning, FSIs responded to the scenes selectively, but were insensitive to object values. In contrast, MSNs became able to discriminate the objects (i.e., stronger response to good objects), but this occurred clearly in one of the two scenes (X or Y). This was caused by the scene-selective inhibition by FSI. As a whole, MSNs were divided into two groups that were sensitive to object values in scene X or in scene Y. These data indicate that the local network of striatum tail controls the learning of object values that are selective to the scene-environment. This mechanism may support our flexible switching behavior in various environments.

Active maintenance of eligibility trace in rodent prefrontal cortex

Even though persistent neural activity has been proposed as a mechanism for maintaining eligibility trace, direct empirical evidence for active maintenance of eligibility trace has been lacking. We recorded neuronal activity in the medial prefrontal cortex (mPFC) in rats performing a dynamic foraging task in which a choice must be remembered until its outcome on the timescale of seconds for correct credit assignment. We found that mPFC neurons maintain significant choice signals during the time period between action selection and choice outcome. We also found that neural signals for choice, outcome, and action value converge in the mPFC when choice outcome was revealed. Our results indicate that the mPFC maintains choice signals necessary for temporal credit assignment in the form of persistent neural activity in our task. They also suggest that the mPFC might update action value by combining actively maintained eligibility trace with action value and outcome signals.

Time as the fourth dimension in the hippocampus

Experiences of animal and human beings are structured by the continuity of space and time coupled with the uni-directionality of time. In addition to its pivotal position in spatial processing and navigation, the hippocampal system also plays a central, multiform role in several types of temporal processing. These include timing and sequence learning, at scales ranging from meso-scales of seconds to macro-scales of minutes, hours, days and beyond, encompassing the classical functions of short term memory, working memory, long term memory, and episodic memories (comprised of information about when, what, and where). This review article highlights the principal findings and behavioral contexts of experiments in rats showing: 1) timing: tracking time during delays by hippocampal 'time cells' and during free behavior by hippocampal-afferent lateral entorhinal cortex ramping cells; 2) 'online' sequence processing: activity coding sequences of events during active behavior; 3) 'off-line' sequence replay: during quiescence or sleep, orderly reactivation of neuronal assemblies coding awake sequences. Studies in humans show neurophysiological correlates of episodic memory comparable to awake replay. Neural mechanisms are discussed, including ion channel properties, plateau and ramping potentials, oscillations of excitation and inhibition of population activity, bursts of high amplitude discharges (sharp wave ripples), as well as short and long term synaptic modifications among and within cell assemblies. Specifically conceived neural network models will suggest processes supporting the emergence of scalar properties (Weber's law), and include different classes of feedforward and recurrent network models, with intrinsic hippocampal coding for 'transitions' (sequencing of events or places).

Functional Dissection of Basal Ganglia Inhibitory Inputs onto Substantia Nigra Dopaminergic Neurons

Substantia nigra (SNc) dopaminergic neurons respond to aversive stimuli with inhibitory pauses in firing followed by transient rebound activation. We tested integration of inhibitory synaptic inputs onto SNc neurons from genetically defined populations in dorsal striatum (striosome and matrix) and external globus pallidus (GPe; parvalbumin- and Lhx6-positive), and examined their contribution to pause-rebound firing. Activation of striosome projections, which target "dendron bouquets" in the pars reticulata (SNr), consistently quiets firing and relief from striosome inhibition triggers rebound activity. Striosomal inhibitory postsynaptic currents (IPSCs) display a prominent GABA-B receptor-mediated component that strengthens the impact of SNr dendrite synapses on somatic excitability and enables rebounding. By contrast, GPe projections activate GABA-A receptors on the soma and proximal dendrites but do not result in rebounding. Lastly, optical mapping shows that dorsal striatum selectively inhibits the ventral population of SNc neurons, which are intrinsically capable of rebounding. Therefore, we define a distinct striatonigral circuit for generating dopamine rebound.

Signaling models for dopamine-dependent temporal contiguity in striatal synaptic plasticity

Animals remember temporal links between their actions and subsequent rewards. We previously discovered a synaptic mechanism underlying such reward learning in D1 receptor (D1R)-expressing spiny projection neurons (D1 SPN) of the striatum. Dopamine (DA) bursts promote dendritic spine enlargement in a time window of only a few seconds after paired pre- and post-synaptic spiking (pre-post pairing), which is termed as reinforcement plasticity (RP). The previous study has also identified underlying signaling pathways; however, it still remains unclear how the signaling dynamics results in RP. In the present study, we first developed a computational model of signaling dynamics of D1 SPNs. The D1 RP model successfully reproduced experimentally observed protein kinase A (PKA) activity, including its critical time window. In this model, adenylate cyclase type 1 (AC1) in the spines/thin dendrites played a pivotal role as a coincidence detector against pre-post pairing and DA burst. In particular, pre-post pairing (Ca2+ signal) stimulated AC1 with a delay, and the Ca2+-stimulated AC1 was activated by the DA burst for the asymmetric time window. Moreover, the smallness of the spines/thin dendrites is crucial to the short time window for the PKA activity. We then developed a RP model for D2 SPNs, which also predicted the critical time window for RP that depended on the timing of pre-post pairing and phasic DA dip. AC1 worked for the coincidence detector in the D2 RP model as well. We further simulated the signaling pathway leading to Ca2+/calmodulin-dependent protein kinase II (CaMKII) activation and clarified the role of the downstream molecules of AC1 as the integrators that turn transient input signals into persistent spine enlargement. Finally, we discuss how such timing windows guide animals' reward learning.

The Meaning of Behavior: Discriminating Reflex and Volition in the Brain

The ability to establish behaviorally what psychological capacity an animal is deploying-to discern accurately what an animal is doing-is key to functional analyses of the brain. Our current understanding of these capacities suggests, however, that this task is complex; there is evidence that multiple capacities are engaged simultaneously and contribute independently to the control of behavior. As such, establishing the contribution of a cell, circuit, or neural system to any one function requires careful dissection of that role from its influence on other functions and, therefore, the careful selection and design of behavioral tasks fit for that purpose. Here I describe recent research that has sought to utilize behavioral tools to investigate the neural bases of instrumental conditioning, particularly the circuits and systems supporting the capacity for goal-directed action, as opposed to conditioned reflexes and habits, and how these sources of action control interact to generate adaptive behavior.

Learning from Action: Reconsidering Movement Signaling in Midbrain Dopamine Neuron Activity

Animals infer when and where a reward is available from experience with informative sensory stimuli and their own actions. In vertebrates, this is thought to depend upon the release of dopamine from midbrain dopaminergic neurons. Studies of the role of dopamine have focused almost exclusively on their encoding of informative sensory stimuli; however, many dopaminergic neurons are active just prior to movement initiation, even in the absence of sensory stimuli. How should current frameworks for understanding the role of dopamine incorporate these observations? To address this question, we review recent anatomical and functional evidence for action-related dopamine signaling. We conclude by proposing a framework in which dopaminergic neurons encode subjective signals of action initiation to solve an internal credit assignment problem.

Synaptic Plasticity Forms and Functions

Synaptic plasticity, the activity-dependent change in neuronal connection strength, has long been considered an important component of learning and memory. Computational and engineering work corroborate the power of learning through the directed adjustment of connection weights. Here we review the fundamental elements of four broadly categorized forms of synaptic plasticity and discuss their functional capabilities and limitations. Although standard, correlation-based, Hebbian synaptic plasticity has been the primary focus of neuroscientists for decades, it is inherently limited. Three-factor plasticity rules supplement Hebbian forms with neuromodulation and eligibility traces, while true supervised types go even further by adding objectives and instructive signals. Finally, a recently discovered hippocampal form of synaptic plasticity combines the above elements, while leaving behind the primary Hebbian requirement. We suggest that the effort to determine the neural basis of adaptive behavior could benefit from renewed experimental and theoretical investigation of more powerful directed types of synaptic plasticity.

Basal ganglia role in learning rewarded actions and executing previously learned choices: Healthy and diseased states

The basal ganglia (BG) is a collection of nuclei located deep beneath the cerebral cortex that is involved in learning and selection of rewarded actions. Here, we analyzed BG mechanisms that enable these functions. We implemented a rate model of a BG-thalamo-cortical loop and simulated its performance in a standard action selection task. We have shown that potentiation of corticostriatal synapses enables learning of a rewarded option. However, these synapses became redundant later as direct connections between prefrontal and premotor cortices (PFC-PMC) were potentiated by Hebbian learning. After we switched the reward to the previously unrewarded option (reversal), the BG was again responsible for switching to the new option. Due to the potentiated direct cortical connections, the system was biased to the previously rewarded choice, and establishing the new choice required a greater number of trials. Guided by physiological research, we then modified our model to reproduce pathological states of mild Parkinson's and Huntington's diseases. We found that in the Parkinsonian state PMC activity levels become extremely variable, which is caused by oscillations arising in the BG-thalamo-cortical loop. The model reproduced severe impairment of learning and predicted that this is caused by these oscillations as well as a reduced reward prediction signal. In the Huntington state, the potentiation of the PFC-PMC connections produced better learning, but altered BG output disrupted expression of the rewarded choices. This resulted in random switching between rewarded and unrewarded choices resembling an exploratory phase that never ended. Along with other computational studies, our results further reconcile the apparent contradiction between the critical involvement of the BG in execution of previously learned actions and yet no impairment of these actions after BG output is ablated by lesions or deep brain stimulation. We predict that the cortico-BG-thalamo-cortical loop conforms to previously learned choice in healthy conditions, but impedes those choices in disease states.

NMDA receptor-dependent plasticity in the nucleus accumbens connects reward-predictive cues to approach responses

Learning associations between environmental cues and rewards is a fundamental adaptive function. Via such learning, reward-predictive cues come to activate approach to locations where reward is available. The nucleus accumbens (NAc) is essential for cued approach behavior in trained subjects, and cue-evoked excitations in NAc neurons are critical for the expression of this behavior. Excitatory synapses within the NAc undergo synaptic plasticity that presumably contributes to cued approach acquisition, but a direct link between synaptic plasticity within the NAc and the development of cue-evoked neural activity during learning has not been established. Here we show that, with repeated cue-reward pairings, cue-evoked excitations in the NAc emerge and grow in the trials prior to the detectable expression of cued approach behavior. We demonstrate that the growth of these signals requires NMDA receptor-dependent plasticity within the NAc, revealing a neural mechanism by which the NAc participates in learning of conditioned reward-seeking behaviors.

Conditioned stimuli elicit phasic changes in nucleus accumbens (NAc) firing that invigorate approach responses to predicted rewards. Here the authors show that NAc neurons acquire cue-evoked responses during learning as a result of excitatory plasticity within the NAc.

Reward-driven changes in striatal pathway competition shape evidence evaluation in decision-making

Cortico-basal-ganglia-thalamic (CBGT) networks are critical for adaptive decision-making, yet how changes to circuit-level properties impact cognitive algorithms remains unclear. Here we explore how dopaminergic plasticity at corticostriatal synapses alters competition between striatal pathways, impacting the evidence accumulation process during decision-making. Spike-timing dependent plasticity simulations showed that dopaminergic feedback based on rewards modified the ratio of direct and indirect corticostriatal weights within opposing action channels. Using the learned weight ratios in a full spiking CBGT network model, we simulated neural dynamics and decision outcomes in a reward-driven decision task and fit them with a drift diffusion model. Fits revealed that the rate of evidence accumulation varied with inter-channel differences in direct pathway activity while boundary height varied with overall indirect pathway activity. This multi-level modeling approach demonstrates how complementary learning and decision computations can emerge from corticostriatal plasticity.

Cognitive process models such as reinforcement learning (RL) and the drift diffusion model (DDM) have helped to elucidate the basic algorithms underlying error-corrective learning and the evaluation of accumulating decision evidence leading up to a choice. While these relatively abstract models help to guide experimental and theoretical probes into associated phenomena, they remain uninformative about the actual physical mechanics by which learning and decision algorithms are carried out in a neurobiological substrate during adaptive choice behavior. Here we present an “upwards mapping” approach to bridging neural and cognitive models of value-based decision-making, showing how dopaminergic feedback alters the network-level dynamics of cortico-basal-ganglia-thalamic (CBGT) pathways during learning to bias behavioral choice towards more rewarding actions. By mapping “up” the levels of analysis, this approach yields specific predictions about aspects of neuronal activity that map to the quantities appearing in the cognitive decision-making framework.