A neurocomputational account of catalepsy sensitization induced by D2 receptor blockade in rats: context dependency, extinction, and renewal

Dopamine D2 receptors in discrimination learning and spine enlargement

Dopamine D2 receptors (D2Rs) are densely expressed in the striatum and have been linked to neuropsychiatric disorders such as schizophrenia^1,2. High-affinity binding of dopamine suggests that D2Rs detect transient reductions in dopamine concentration (the dopamine dip) during punishment learning^3-5. However, the nature and cellular basis of D2R-dependent behaviour are unclear. Here we show that tone reward conditioning induces marked stimulus generalization in a manner that depends on dopamine D1 receptors (D1Rs) in the nucleus accumbens (NAc) of mice, and that discrimination learning refines the conditioning using a dopamine dip. In NAc slices, a narrow dopamine dip (as short as 0.4 s) was detected by D2Rs to disinhibit adenosine A_2A receptor (A_2AR)-mediated enlargement of dendritic spines in D2R-expressing spiny projection neurons (D2-SPNs). Plasticity-related signalling by Ca²⁺/calmodulin-dependent protein kinase II and A_2ARs in the NAc was required for discrimination learning. By contrast, extinction learning did not involve dopamine dips or D2-SPNs. Treatment with methamphetamine, which dysregulates dopamine signalling, impaired discrimination learning and spine enlargement, and these impairments were reversed by a D2R antagonist. Our data show that D2Rs refine the generalized reward learning mediated by D1Rs.

Revisiting anticipatory hedonic processing in patients with schizophrenia: An examination between representation activation and maintenance

BACKGROUND

Anticipatory anhedonia is one of the key deficits found in patients with schizophrenia (SCZ). However, the underlying mechanism of this deficit remains unclear. The present study examined whether representation activation and maintenance capacity influenced anticipatory experiences in SCZ patients.

METHODS

We recruited 46 SCZ patients (26 males) and 45 matched healthy controls (24 males). The Reward Representation Activation and Maintenance (RRAM) Task was administrated to assess anticipatory experience and representation activation and maintenance capacity.

RESULTS

SCZ patients exhibited lower subjective arousal than controls in anticipation of rewards with high probability when representation activation and maintenance were difficult to accomplish. SCZ patients also tended to reduce their button presses more than HC when they were required to maintain reward representation.

CONCLUSIONS

Our findings suggest that representation activation and maintenance may partially account for anticipatory anhedonia observed in SCZ patients.

Dopamine D2 -141C Ins/Del and Taq1A polymorphisms, body mass index, and prediction error brain response

The prediction error model is a widely used paradigm that is conceptually based on neuronal dopamine function. However, whether dopamine receptor gene alleles contribute to human neuroimaging prediction error results is uncertain. Recent research implicated the dopamine D2 receptor in behavior response during a prediction error paradigm and we expected that polymorphisms of that receptor would contribute to prediction error brain response. In this study, healthy female participants in the early follicular phase of the menstrual cycle underwent a taste prediction error paradigm during functional magnetic resonance imaging. Participants were also genotyped for dopamine receptor polymorphisms. Our data suggest that the dopamine D2 receptor -141C Ins/Del and Taq1A polymorphisms together with body mass index selectively explain putamen prediction error response. This was true using a region of interest analysis as well as for a whole-brain analysis (FWE corrected). Polymorphisms for dopamine D1 or D4 receptors, dopamine transporter, or COMT did not significantly contribute to prediction error activation. The prediction error model is a computational reward-learning paradigm that is important in psychiatric research and has been associated with dopamine. The results from this study indicate that dopamine D2 receptor polymorphisms together with body mass index are important determinants to include in research that tests prediction error response of the brain. Psychiatric disorders are frequently associated with elevated or reduced body weight. Adding BMI to genetic information in brain-imaging studies that use reward and the prediction error paradigm may be important to increase validity and reliability of results.

To Do or Not to Do: Dopamine, Affordability and the Economics of Opportunity

Five years ago, we introduced the thrift hypothesis of dopamine (DA), suggesting that the primary role of DA in adaptive behavior is regulating behavioral energy expenditure to match the prevailing economic conditions of the environment. Here we elaborate that hypothesis with several new ideas. First, we introduce the concept of affordability, suggesting that costs must necessarily be evaluated with respect to the availability of resources to the organism, which computes a value not only for the potential reward opportunity, but also the value of resources expended. Placing both costs and benefits within the context of the larger economy in which the animal is functioning requires consideration of the different timescales against which to compute resource availability, or average reward rate. Appropriate windows of computation for tracking resources requires corresponding neural substrates that operate on these different timescales. In discussing temporal patterns of DA signaling, we focus on a neglected form of DA plasticity and adaptation, changes in the physical substrate of the DA system itself, such as up- and down-regulation of receptors or release probability. We argue that changes in the DA substrate itself fundamentally alter its computational function, which we propose mediates adaptations to longer temporal horizons and economic conditions. In developing our hypothesis, we focus on DA D2 receptors (D2R), arguing that D2R implements a form of “cost control” in response to the environmental economy, serving as the “brain’s comptroller”. We propose that the balance between the direct and indirect pathway, regulated by relative expression of D1 and D2 DA receptors, implements affordability. Finally, as we review data, we discuss limitations in current approaches that impede fully investigating the proposed hypothesis and highlight alternative, more semi-naturalistic strategies more conducive to neuroeconomic investigations on the role of DA in adaptive behavior.

Interrelations between cognitive dysfunction and motor symptoms of Parkinson's disease: behavioral and neural studies

Parkinson's disease (PD) is characterized by a range of motor symptoms. Besides the cardinal symptoms (tremor, bradykinesia/akinesia, and rigidity), PD patients also show other motor deficits, including gait disturbance, speech deficits, and impaired handwriting. However, along with these key motor symptoms, PD patients also experience cognitive deficits in attention, executive function, working memory, and learning. Recent evidence suggests that these motor and cognitive deficits of PD are not completely dissociable, as aspects of cognitive dysfunction can impact motor performance in PD. In this article, we provide a review of behavioral and neural studies on the associations between motor symptoms and cognitive deficits in PD, specifically akinesia/bradykinesia, tremor, gait, handwriting, precision grip, and speech production. This review paves the way for providing a framework for understanding how treatment of cognitive dysfunction, for example cognitive rehabilitation programs, may in turn influence the motor symptoms of PD.

Chronic Nicotine Mitigates Aberrant Inhibitory Motor Learning Induced by Motor Experience under Dopamine Deficiency

Although dopamine receptor antagonism has long been associated with impairments in motor performance, more recent studies have shown that dopamine D2 receptor (D2R) antagonism, paired with a motor task, not only impairs motor performance concomitant with the pharmacodynamics of the drug, but also impairs future motor performance once antagonism has been relieved. We have termed this phenomenon "aberrant motor learning" and have suggested that it may contribute to motor symptoms in movement disorders such as Parkinson's disease (PD). Here, we show that chronic nicotine (cNIC), but not acute nicotine, treatment mitigates the acquisition of D2R-antagonist-induced aberrant motor learning in mice. Although cNIC mitigates D2R-mediated aberrant motor learning, cNIC has no effect on D1R-mediated motor learning. β2-containing nicotinic receptors in dopamine neurons likely mediate the protective effect of cNIC against aberrant motor learning, because selective deletion of β2 nicotinic subunits in dopamine neurons reduced D2R-mediated aberrant motor learning. Finally, both cNIC treatment and β2 subunit deletion blunted postsynaptic responses to D2R antagonism. These results suggest that a chronic decrease in function or a downregulation of β2-containing nicotinic receptors protects the striatal network against aberrant plasticity and aberrant motor learning induced by motor experience under dopamine deficiency.

SIGNIFICANCE STATEMENT

Increasingly, aberrant plasticity and aberrant learning are recognized as contributing to the development and progression of movement disorders. Here, we show that chronic nicotine (cNIC) treatment or specific deletion of β2 nicotinic receptor subunits in dopamine neurons mitigates aberrant motor learning induced by dopamine D2 receptor (D2R) blockade in mice. Moreover, both manipulations also reduced striatal dopamine release and blunt postsynaptic responses to D2R antagonists. These results suggest that chronic downregulation of function and/or receptor expression of β2-containing nicotinic receptors alters presynaptic and postsynaptic striatal signaling to protect against aberrant motor learning. Moreover, these results suggest that cNIC treatment may alleviate motor symptoms and/or delay the deterioration of motor function in movement disorders by blocking aberrant motor learning.

Short- and long-term effects of risperidone on catalepsy sensitisation and acquisition of conditioned avoidance response: Adolescent vs adult rats

The effects of antipsychotic drugs (APDs) on the adolescent brain are poorly understood despite a dramatic increase in prescription of these drugs in adolescents over the past twenty years. Neuronal systems continue to be remodeled during adolescence. Therefore, when given in adolescence, antipsychotic drugs (APDs) have the potential to affect this remodeling. In this study we investigated the effects of chronic 22-day risperidone treatment (1.3mg/kg/day) in both adolescent and adult rats. We examined short- and long-term changes in behaviour (catalepsy, locomotion and conditioned avoidance response (CAR)), and dopaminergic and serotonergic neurochemistry in the striatum and the nucleus accumbens. Here, we report that, both during chronic treatment and after a lengthy drug-free interval, risperidone induced a sensitised cataleptic response regardless of the age of exposure. Selectively in adolescents, risperidone-induced catalepsy was inversely correlated with striatal dopamine turnover immediately after chronic treatment. After a drug-free interval, a significant proportion of rats with prior adolescent risperidone treatment also failed to acquire CAR to a defined criterion. Our data provide evidence that the same chronic risperidone treatment regimen can induce contrasting short- and long-term neural outcomes in the adolescent and adult brains.

An Integrative Perspective on the Role of Dopamine in Schizophrenia

We propose that schizophrenia involves a combination of decreased phasic dopamine responses for relevant stimuli and increased spontaneous phasic dopamine release. Using insights from computational reinforcement-learning models and basic-science studies of the dopamine system, we show that each of these two disturbances contributes to a specific symptom domain and explains a large set of experimental findings associated with that domain. Reduced phasic responses for relevant stimuli help to explain negative symptoms and provide a unified explanation for the following experimental findings in schizophrenia, most of which have been shown to correlate with negative symptoms: reduced learning from rewards; blunted activation of the ventral striatum, midbrain, and other limbic regions for rewards and positive prediction errors; blunted activation of the ventral striatum during reward anticipation; blunted autonomic responding for relevant stimuli; blunted neural activation for aversive outcomes and aversive prediction errors; reduced willingness to expend effort for rewards; and psychomotor slowing. Increased spontaneous phasic dopamine release helps to explain positive symptoms and provides a unified explanation for the following experimental findings in schizophrenia, most of which have been shown to correlate with positive symptoms: aberrant learning for neutral cues (assessed with behavioral and autonomic responses), and aberrant, increased activation of the ventral striatum, midbrain, and other limbic regions for neutral cues, neutral outcomes, and neutral prediction errors. Taken together, then, these two disturbances explain many findings in schizophrenia. We review evidence supporting their co-occurrence and consider their differential implications for the treatment of positive and negative symptoms.

Motivational Context Modulates Prediction Error Response in Schizophrenia

BACKGROUND

Recent findings demonstrate that patients with schizophrenia are worse at learning to predict rewards than losses, suggesting that motivational context modulates learning in this disease. However, these findings derive from studies in patients treated with antipsychotic medications, D2 receptor antagonists that may interfere with the neural systems that underlie motivation and learning. Thus, it remains unknown how motivational context affects learning in schizophrenia, separate from the effects of medication.

METHODS

To examine the impact of motivational context on learning in schizophrenia, we tested 16 unmedicated patients with schizophrenia and 23 matched controls on a probabilistic learning task while they underwent functional magnetic resonance imaging (fMRI) under 2 conditions: one in which they pursued rewards, and one in which they avoided losses. Computational models were used to derive trial-by-trial prediction error responses to feedback.

RESULTS

Patients performed worse than controls on the learning task overall, but there were no behavioral effects of condition. FMRI revealed an attenuated prediction error response in patients in the medial prefrontal cortex, striatum, and medial temporal lobe when learning to predict rewards, but not when learning to avoid losses.

CONCLUSIONS

Patients with schizophrenia showed differences in learning-related brain activity when learning to predict rewards, but not when learning to avoid losses. Together with prior work, these results suggest that motivational deficits related to learning in schizophrenia are characteristic of the disease and not solely a result of antipsychotic treatment.

DYT1 dystonia increases risk taking in humans

It has been difficult to link synaptic modification to overt behavioral changes. Rodent models of DYT1 dystonia, a motor disorder caused by a single gene mutation, demonstrate increased long-term potentiation and decreased long-term depression in corticostriatal synapses. Computationally, such asymmetric learning predicts risk taking in probabilistic tasks. Here we demonstrate abnormal risk taking in DYT1 dystonia patients, which is correlated with disease severity, thereby supporting striatal plasticity in shaping choice behavior in humans.

DOI:http://dx.doi.org/10.7554/eLife.14155.001

We learn to choose better options and avoid worse ones through trial and error, but exactly how this happens is still unclear. One idea is that we learn 'values' for options: whenever we choose an option and get more reward than originally expected (for example, if an unappetizing-looking food turns out to be very tasty), the value of that option increases. Likewise, if we get less reward than expected, the chosen option’s value decreases.

This learning process is hypothesized to work via the strengthening and weakening of connections between neurons in two parts of the brain: the cortex and the striatum. In this model, the activity of the neurons in the cortex represents the options, and the value of these options is represented by the activity of neurons in the striatum. Strengthening the connections is thought to increase the value of the stimulus, but this theory has been difficult to test.

In humans, a single genetic mutation causes a movement disorder called DYT1 dystonia, in which muscles contract involuntarily. In rodents, the same mutation causes the connections between the neurons in the cortex and the striatum to become too strong. If the theory about value learning is true, this strengthening should affect the decisions of patients that have DYT1 dystonia.

Arkadir et al. got healthy people and people with DYT1 dystonia to play a game where they had to choose between a 'sure' option and a 'risky' option. Picking the sure option guaranteed the player would receive a small amount of money, whereas the risky option gave either double this amount or nothing. The theory predicts that the double rewards should cause the patients to learn abnormally high values, which would lure them into making risky choices. Indeed, Arkadir et al. found that players with DYT1 dystonia were more likely to choose the risky option, with the people who had more severe symptoms of dystonia having a greater tendency towards taking risks.

Arkadir et al. showed that these results correspond with a model that suggests that people with DYT1 dystonia learn excessively from unexpected wins but show weakened learning after losses, causing them to over-estimate the value of risky choices. This imbalance mirrors the previous results that showed an inappropriate strengthening of the connections between neurons in rodents, and so suggests that similar changes occur in the brains of humans. Thus it appears that the changes in the strength of the connections between neurons translate into changes in behavior.

This pattern of results might also mean that the movement problems seen in people with DYT1 dystonia may be because they over-learn movements that previously led to a desired outcome and cannot sufficiently suppress movements that are no longer useful. Testing this idea will require further experiments.

DOI:http://dx.doi.org/10.7554/eLife.14155.002

Rethinking psychopharmacotherapy: The role of treatment context and brain plasticity in antidepressant and antipsychotic interventions

Emerging evidence indicates that treatment context profoundly affects psychopharmacological interventions. We review the evidence for the interaction between drug application and the context in which the drug is given both in human and animal research. We found evidence for this interaction in the placebo response in clinical trials, in our evolving knowledge of pharmacological and environmental effects on neural plasticity, and in animal studies analyzing environmental influences on psychotropic drug effects. Experimental placebo research has revealed neurobiological trajectories of mechanisms such as patients' treatment expectations and prior treatment experiences. Animal research confirmed that "enriched environments" support positive drug effects, while unfavorable environments (low sensory stimulation, low rates of social contacts) can even reverse the intended treatment outcome. Finally we provide recommendations for context conditions under which psychotropic drugs should be applied. Drug action should be steered by positive expectations, physical activity, and helpful social and physical environmental stimulation. Future drug trials should focus on fully controlling and optimizing such drug×environment interactions to improve trial sensitivity and treatment outcome.

A computational analysis of flanker interference in depression

BACKGROUND

Depression is characterized by poor executive function, but - counterintuitively - in some studies, it has been associated with highly accurate performance on certain cognitively demanding tasks. The psychological mechanisms responsible for this paradoxical finding are unclear. To address this issue, we applied a drift diffusion model (DDM) to flanker task data from depressed and healthy adults participating in the multi-site Establishing Moderators and Biosignatures of Antidepressant Response for Clinical Care for Depression (EMBARC) study.

METHOD

One hundred unmedicated, depressed adults and 40 healthy controls completed a flanker task. We investigated the effect of flanker interference on accuracy and response time, and used the DDM to examine group differences in three cognitive processes: prepotent response bias (tendency to respond to the distracting flankers), response inhibition (necessary to resist prepotency), and executive control (required for execution of correct response on incongruent trials).

RESULTS

Consistent with prior reports, depressed participants responded more slowly and accurately than controls on incongruent trials. The DDM indicated that although executive control was sluggish in depressed participants, this was more than offset by decreased prepotent response bias. Among the depressed participants, anhedonia was negatively correlated with a parameter indexing the speed of executive control (r = -0.28, p = 0.007).

CONCLUSIONS

Executive control was delayed in depression but this was counterbalanced by reduced prepotent response bias, demonstrating how participants with executive function deficits can nevertheless perform accurately in a cognitive control task. Drawing on data from neural network simulations, we speculate that these results may reflect tonically reduced striatal dopamine in depression.

Cyclic AMP and afferent activity govern bidirectional synaptic plasticity in striatopallidal neurons

Recent experimental evidence suggests that the low dopamine conditions in Parkinson's disease (PD) cause motor impairment through aberrant motor learning. Those data, along with computational models, suggest that this aberrant learning results from maladaptive corticostriatal plasticity and learned motor inhibition. Dopaminergic modulation of both corticostriatal long-term depression (LTD) and long-term potentiation (LTP) is proposed to be critical for these processes; however, the regulatory mechanisms underlying bidirectional corticostriatal plasticity are not fully understood. Previously, we demonstrated a key role for cAMP signaling in corticostriatal LTD. In this study, mouse brain slices were used to perform a parametric experiment that tested the impact of varying both intracellular cAMP levels and the strength of excitatory inputs on corticostriatal plasticity. Using slice electrophysiology in the dorsolateral striatum, we demonstrate that both LTP and LTD can be sequentially induced in the same D2-expressing neuron and that LTP was strongest with high intracellular cAMP and LFS, whereas LTD required low intracellular cAMP and high-frequency stimulation. Our results provide a molecular and cellular basis for regulating bidirectional corticostriatal synaptic plasticity and may help to identify novel therapeutic targets for blocking or reversing the aberrant synaptic plasticity that likely contributes to motor deficits in PD.

Cognitive control over learning: creating, clustering, and generalizing task-set structure

Learning and executive functions such as task-switching share common neural substrates, notably prefrontal cortex and basal ganglia. Understanding how they interact requires studying how cognitive control facilitates learning but also how learning provides the (potentially hidden) structure, such as abstract rules or task-sets, needed for cognitive control. We investigate this question from 3 complementary angles. First, we develop a new context-task-set (C-TS) model, inspired by nonparametric Bayesian methods, specifying how the learner might infer hidden structure (hierarchical rules) and decide to reuse or create new structure in novel situations. Second, we develop a neurobiologically explicit network model to assess mechanisms of such structured learning in hierarchical frontal cortex and basal ganglia circuits. We systematically explore the link between these modeling levels across task demands. We find that the network provides an approximate implementation of high-level C-TS computations, with specific neural mechanisms modulating distinct C-TS parameters. Third, this synergism yields predictions about the nature of human optimal and suboptimal choices and response times during learning and task-switching. In particular, the models suggest that participants spontaneously build task-set structure into a learning problem when not cued to do so, which predicts positive and negative transfer in subsequent generalization tests. We provide experimental evidence for these predictions and show that C-TS provides a good quantitative fit to human sequences of choices. These findings implicate a strong tendency to interactively engage cognitive control and learning, resulting in structured abstract representations that afford generalization opportunities and, thus, potentially long-term rather than short-term optimality.

The Enemy within: Propagation of Aberrant Corticostriatal Learning to Cortical Function in Parkinson's Disease

Motor dysfunction in Parkinson’s disease is believed to arise primarily from pathophysiology in the dorsal striatum and its related corticostriatal and thalamostriatal circuits during progressive dopamine denervation. One function of these circuits is to provide a filter that selectively facilitates or inhibits cortical activity to optimize cortical processing, making motor responses rapid and efficient. Corticostriatal synaptic plasticity mediates the learning that underlies this performance-optimizing filter. Under dopamine denervation, corticostriatal plasticity is altered, resulting in aberrant learning that induces inappropriate basal ganglia filtering that impedes rather than optimizes cortical processing. Human imaging suggests that increased cortical activity may compensate for striatal dysfunction in PD patients. In this Perspective article, we consider how aberrant learning at corticostriatal synapses may impair cortical processing and learning and undermine potential cortical compensatory mechanisms. Blocking or remediating aberrant corticostriatal plasticity may protect cortical function and support cortical compensatory mechanisms mitigating the functional decline associated with progressive dopamine denervation.

The role of neuroplasticity in dopaminergic therapy for Parkinson disease

Dopamine replacement is a mainstay of therapeutic strategies for Parkinson disease (PD). The motor response to therapy involves an immediate improvement in motor function, known as the short-duration response (SDR), followed by a long-duration response (LDR) that develops more slowly, over weeks. Here, we review evidence in patients and animal models suggesting that dopamine-dependent corticostriatal plasticity, and retention of such plasticity in the absence of dopamine, are the mechanisms underlying the LDR. Conversely, experience-dependent aberrant plasticity that develops slowly under reduced dopamine levels could contribute substantially to PD motor symptoms before initiation of dopamine replacement therapy. We place these findings in the context of the role of dopamine in basal ganglia function and corticostriatal plasticity, and provide a new framework suggesting that therapies that enhance the LDR could be more effective than those targeting the SDR. We further propose that changes in neuroplasticity constitute a form of disease modification that is distinct from prevention of degeneration, and could be responsible for some of the unexplained disease-modifying effects of certain therapies. Understanding such plasticity could provide novel therapeutic approaches that combine rehabilitation and pharmacotherapy for treatment of neurological and psychiatric disorders involving basal ganglia dysfunction.

A role for dopamine-mediated learning in the pathophysiology and treatment of Parkinson's disease

Dopamine contributes to corticostriatal plasticity and motor learning. Dopamine denervation profoundly alters motor performance, as in Parkinson's disease (PD); however, the extent to which these symptoms reflect impaired motor learning is unknown. Here, we demonstrate a D2 receptor blockade-induced aberrant learning that impedes future motor performance when dopamine signaling is restored, an effect diminished by coadministration of adenosine antagonists during blockade. We hypothesize that an inappropriate corticostriatal potentiation in striatopallidal cells of the indirect pathway underlies aberrant learning. We demonstrate synaptic potentiation in striatopallidal neurons induced by D2 blockade and diminished by application of an adenosine antagonist, consistent with behavioral observations. A neurocomputational model of the basal ganglia recapitulates the behavioral pattern and further links aberrant learning to plasticity in the indirect pathway. Thus, D2-mediated aberrant learning may contribute to motor deficits in PD, suggesting new avenues for the development of therapeutics.

Haloperidol conditioned catalepsy in rats: a possible role for D1-like receptors

Decreases in brain dopamine (DA) lead to catalepsy, quantified by the time a rat remains with its forepaws resting on a suspended horizontal bar. Low doses of the DA D2 receptor-preferring antagonist haloperidol repeatedly injected in a particular environment lead to gradual day-to-day increases in catalepsy (catalepsy sensitization) and subsequent testing following an injection of saline reveal conditioned catalepsy. We tested the hypothesis that D1-like and D2 receptors play different roles in catalepsy sensitization and in acquisition and expression of conditioned catalepsy. Rats were repeatedly treated with the DA D1-like receptor antagonist SCH 23990 (0.05, 0.1 and 0.25 mg/kg i.p.), the D2 receptor-preferring antagonist haloperidol (0.1, 0.25 and 0.5 mg/kg i.p.) or a combination of the two drugs and tested for catalepsy each day in the same environment. Following 10 drug treatment days, rats were injected with saline and tested for conditioned catalepsy in the previously drug-paired environment. Haloperidol did not elicit cataleptic responses in the initial session; however, rats developed sensitization with repeated testing. Significant catalepsy sensitization was not observed in rats repeatedly tested with SCH 23390. When rats were injected and tested with saline following haloperidol sensitization they exhibited conditioned catalepsy in the test environment; conditioned catalepsy was not seen following SCH 23390. Rats treated with 0.05 mg/kg SCH 23390+0.25 mg/kg haloperidol showed catalepsy sensitization but failed to show conditioned catalepsy. Conversely, SCH 23390 (0.05 mg/kg) given on the test day after sensitization to haloperidol (0.25 mg/kg) failed to block conditioned catalepsy. Repeated antagonism of D2 receptors leads to catalepsy sensitization with repeated testing in a specific environment. Conditioned catalepsy requires intact D1-like receptor function during sensitization sessions but not during test sessions. In conclusion, repeated antagonism of D2, but not D1-like receptors leads to catalepsy sensitization with repeated testing in a specific environment. Conditioned catalepsy requires functional D1-like receptors during sensitization sessions but not during test sessions.

Reduced expression of haloperidol conditioned catalepsy in rats by the dopamine D3 receptor antagonists nafadotride and NGB 2904

Haloperidol, a dopamine (DA) D2 receptor-preferring antagonist, produces catalepsy whereby animals maintain awkward posture for a period of time. Sub-threshold doses of haloperidol fail to produce catalepsy initially, however, when the drug is given repeatedly in the same test environment, gradual day-to-day increases in catalepsy are observed. More importantly, if sensitized rats are injected with saline instead of haloperidol they continue to be cataleptic in the test environment suggesting that environment-drug associations may play a role. DA D3 receptors have been implicated in a number of conditioned behaviors. We were interested if DA D3 receptors contribute to catalepsy sensitization and conditioning in rats. We tested this hypothesis using the DA D3 receptor-selective antagonist NGB 2904 (0.5, 1.8 mg/kg) and the DA D3 receptor-preferring antagonist nafadotride (0.1, 0.5 mg/kg). For 10 consecutive conditioning days rats were treated with one of the D3 receptor antagonists alone or in combination with haloperidol (0.25 mg/kg) and tested for catalepsy, quantified by the time a rat remained with its forepaws on a horizontal bar. On test day (day 11), rats were injected with saline or the D3 receptor antagonist and tested for conditioned catalepsy in the previously drug-paired environment. Rats treated with NGB 2904 or nafadotride alone did not develop catalepsy. Rats treated with haloperidol or haloperidol plus NGB 2904 or nafadotride developed catalepsy sensitization with repeated conditioning. When injected with saline they continued to exhibit catalepsy in the test environment--now conditioned. On the other hand, NGB 2904 (1.8 mg/kg) or nafadotride (0.5 mg/kg) given on the test day (after sensitization to haloperidol) significantly attenuated the expression of conditioned catalepsy. Our data suggest that the D3 receptor antagonist NGB 2904 (1.8 mg/kg) and nafadotride (0.5 mg/kg) significantly attenuate conditioned catalepsy in rats when given in test but not when given during sensitization. Results implicate DA D3 receptors in regulating the expression of conditioned catalepsy.

Reinforcement-Based Decision Making in Corticostriatal Circuits: Mutual Constraints by Neurocomputational and Diffusion Models

In this letter, we examine the computational mechanisms of reinforce-ment-based decision making. We bridge the gap across multiple levels of analysis, from neural models of corticostriatal circuits—the basal ganglia (BG) model (Frank, 2005 , 2006 ) to simpler but mathematically tractable diffusion models of two-choice decision making. Specifically, we generated simulated data from the BG model and fit the diffusion model (Ratcliff, 1978 ) to it. The standard diffusion model fits underestimated response times under conditions of high response and reinforcement conflict. Follow-up fits showed good fits to the data both by increasing nondecision time and by raising decision thresholds as a function of conflict and by allowing this threshold to collapse with time. This profile captures the role and dynamics of the subthalamic nucleus in BG circuitry, and as such, parametric modulations of projection strengths from this nucleus were associated with parametric increases in decision boundary and its modulation by conflict. We then present data from a human reinforcement learning experiment involving decisions with low- and high-reinforcement conflict. Again, the standard model failed to fit the data, but we found that two variants similar to those that fit the BG model data fit the experimental data, thereby providing a convergence of theoretical accounts of complex interactive decision-making mechanisms consistent with available data. This work also demonstrates how to make modest modifications to diffusion models to summarize core computations of the BG model. The result is a better fit and understanding of reinforcement-based choice data than that which would have occurred with either model alone.

Preservation of function in Parkinson's disease: what's learning got to do with it?

Dopamine denervation gives rise to abnormal corticostriatal plasticity; however, its role in the symptoms and progression of Parkinson's disease (PD) has not been articulated or incorporated into current clinical models. The 'integrative selective gain' framework proposed here integrates dopaminergic mechanisms known to modulate basal ganglia throughput into a single conceptual framework: (1) synaptic weights, the neural instantiation of accumulated experience and skill modulated by dopamine-dependent plasticity and (2) system gain, the operating parameters of the basal ganglia, modulated by dopamine's on-line effects on cell excitability, glutamatergic transmission and the balance between facilitatory and inhibitory pathways. Within this framework and based on recent work, a hypothesis is presented that prior synaptic weights and established skills can facilitate motor performance and preserve function despite diminished dopamine; however, dopamine denervation induces aberrant corticostriatal plasticity that degrades established synaptic weights and replaces them with inappropriate, inhibitory learning that inverts the function of the basal ganglia resulting in 'anti-optimization' of motor performance. Consequently, mitigating aberrant corticostriatal plasticity represents an important therapeutic objective, as reflected in the long-duration response to levodopa, reinterpreted here as the correction of aberrant learning. It is proposed that viewing aberrant corticostriatal plasticity and learning as a provisional endophenotype of PD would facilitate investigation of this hypothesis.

Altered probabilistic learning and response biases in schizophrenia: behavioral evidence and neurocomputational modeling

OBJECTIVE

Patients with schizophrenia (SZ) show reinforcement learning impairments related to both the gradual/procedural acquisition of reward contingencies, and the ability to use trial-to-trial feedback to make rapid behavioral adjustments.

METHOD

We used neurocomputational modeling to develop plausible mechanistic hypotheses explaining reinforcement learning impairments in individuals with SZ. We tested the model with a novel Go/NoGo learning task in which subjects had to learn to respond or withhold responses when presented with different stimuli associated with different probabilities of gains or losses in points. We analyzed data from 34 patients and 23 matched controls, characterizing positive- and negative-feedback-driven learning in both a training phase and a test phase.

RESULTS

Consistent with simulations from a computational model of aberrant dopamine input to the basal ganglia patients, patients with SZ showed an overall increased rate of responding in the training phase, together with reduced response-time acceleration to frequently rewarded stimuli across training blocks, and a reduced relative preference for frequently rewarded training stimuli in the test phase. Patients did not differ from controls on measures of procedural negative-feedback-driven learning, although patients with SZ exhibited deficits in trial-to-trial adjustments to negative feedback, with these measures correlating with negative symptom severity.

CONCLUSIONS

These findings support the hypothesis that patients with SZ have a deficit in procedural "Go" learning, linked to abnormalities in DA transmission at D1-type receptors, despite a "Go bias" (increased response rate), potentially related to excessive tonic dopamine. Deficits in trial-to-trial reinforcement learning were limited to a subset of patients with SZ with severe negative symptoms, putatively stemming from prefrontal cortical dysfunction.

Computational models of motivated action selection in corticostriatal circuits

Computational models of the basal ganglia have matured and received increasing attention over the last decade. This article reviews some of the theoretical advances offered by these models, focusing on motor and cognitive action selection, learning, and the interaction between multiple corticostriatal circuits in selection and learning.

Deficits in positive reinforcement learning and uncertainty-driven exploration are associated with distinct aspects of negative symptoms in schizophrenia

BACKGROUND

Negative symptoms are core features of schizophrenia (SZ); however, the cognitive and neural basis for individual negative symptom domains remains unclear. Converging evidence suggests a role for striatal and prefrontal dopamine in reward learning and the exploration of actions that might produce outcomes that are better than the status quo. The current study examines whether deficits in reinforcement learning and uncertainty-driven exploration predict specific negative symptom domains.

METHODS

We administered a temporal decision-making task, which required trial-by-trial adjustment of reaction time to maximize reward receipt, to 51 patients with SZ and 39 age-matched healthy control subjects. Task conditions were designed such that expected value (probability × magnitude) increased, decreased, or remained constant with increasing response times. Computational analyses were applied to estimate the degree to which trial-by-trial responses are influenced by reinforcement history.

RESULTS

Individuals with SZ showed impaired Go learning but intact NoGo learning relative to control subjects. These effects were most pronounced in patients with higher levels of negative symptoms. Uncertainty-based exploration was substantially reduced in individuals with SZ and selectively correlated with clinical ratings of anhedonia.

CONCLUSIONS

Schizophrenia patients, particularly those with high negative symptoms, failed to speed reaction times to increase positive outcomes and showed reduced tendency to explore when alternative actions could lead to better outcomes than the status quo. Results are interpreted in the context of current computational, genetic, and pharmacological data supporting the roles of striatal and prefrontal dopamine in these processes.

Neurogenetics and pharmacology of learning, motivation, and cognition

Many of the individual differences in cognition, motivation, and learning-and the disruption of these processes in neurological conditions-are influenced by genetic factors. We provide an integrative synthesis across human and animal studies, focusing on a recent spate of evidence implicating a role for genes controlling dopaminergic function in frontostriatal circuitry, including COMT, DARPP-32, DAT1, DRD2, and DRD4. These genetic effects are interpreted within theoretical frameworks developed in the context of the broader cognitive and computational neuroscience literature, constrained by data from pharmacological, neuroimaging, electrophysiological, and patient studies. In this framework, genes modulate the efficacy of particular neural computations, and effects of genetic variation are revealed by assays designed to be maximally sensitive to these computations. We discuss the merits and caveats of this approach and outline a number of novel candidate genes of interest for future study.

Dopamine-dependent motor learning: insight into levodopa's long-duration response

OBJECTIVE

Dopamine (DA) is critical for motor performance, motor learning, and corticostriatal plasticity. The relationship between motor performance and learning, and the role of DA in the mediation of them, however, remain unclear.

METHODS

To examine this question, we took advantage of PITx3-deficient mice (aphakia mice), in which DA in the dorsal striatum is reduced by 90%. PITx3-deficient mice do not display obvious motor deficits in their home cage, but are impaired in motor tasks that require new motor skills. We used the accelerating rotarod as a motor learning task.

RESULTS

We show that the deficiency in motor skill learning in PITx3(-/-) is dramatic and can be rescued with levodopa treatment. In addition, cessation of levodopa treatment after acquisition of the motor skill does not result in an immediate drop in performance. Instead, there is a gradual decline of performance that lasts for a few days, which is not related to levodopa pharmacokinetics. We show that this gradual decline is dependent on the retesting experience.

INTERPRETATION

This observation resembles the long-duration response to levodopa therapy in its slow buildup of improvement after the initiation of therapy and gradual degradation. We hypothesize that motor learning may play a significant, underappreciated role in the symptomatology of Parkinson disease as well as in the therapeutic effects of levodopa. We suggest that the important, yet enigmatic long-duration response to chronic levodopa treatment is a manifestation of rescued motor learning.

Computational models of reinforcement learning: the role of dopamine as a reward signal

Reinforcement learning is ubiquitous. Unlike other forms of learning, it involves the processing of fast yet content-poor feedback information to correct assumptions about the nature of a task or of a set of stimuli. This feedback information is often delivered as generic rewards or punishments, and has little to do with the stimulus features to be learned. How can such low-content feedback lead to such an efficient learning paradigm? Through a review of existing neuro-computational models of reinforcement learning, we suggest that the efficiency of this type of learning resides in the dynamic and synergistic cooperation of brain systems that use different levels of computations. The implementation of reward signals at the synaptic, cellular, network and system levels give the organism the necessary robustness, adaptability and processing speed required for evolutionary and behavioral success.

Neurocomputational models of motor and cognitive deficits in Parkinson's disease

We review the contributions of biologically constrained computational models to our understanding of motor and cognitive deficits in Parkinson's disease (PD). The loss of dopaminergic neurons innervating the striatum in PD, and the well-established role of dopamine (DA) in reinforcement learning (RL), enable neural network models of the basal ganglia (BG) to derive concrete and testable predictions. We focus in this review on one simple underlying principle - the notion that reduced DA increases activity and causes long-term potentiation in the indirect pathway of the BG. We show how this theory can provide a unified account of diverse and seemingly unrelated phenomena in PD including progressive motor degeneration as well as cognitive deficits in RL, decision making and working memory. DA replacement therapy and deep brain stimulation can alleviate some aspects of these impairments, but can actually introduce negative effects such as motor dyskinesias and cognitive impulsivity. We discuss these treatment effects in terms of modulation of specific mechanisms within the computational framework. In addition, we review neurocomputational interpretations of increased impulsivity in the face of response conflict in patients with deep-brain-stimulation.

Prefrontal and striatal dopaminergic genes predict individual differences in exploration and exploitation

The basal ganglia support learning to exploit decisions that have yielded positive outcomes in the past. In contrast, limited evidence implicates the prefrontal cortex in the process of making strategic exploratory decisions when the magnitude of potential outcomes is unknown. Here we examine neurogenetic contributions to individual differences in these distinct aspects of motivated human behavior, using a temporal decision-making task and computational analysis. We show that two genes controlling striatal dopamine function, DARPP-32 (also called PPP1R1B) and DRD2, are associated with exploitative learning to adjust response times incrementally as a function of positive and negative decision outcomes. In contrast, a gene primarily controlling prefrontal dopamine function (COMT) is associated with a particular type of 'directed exploration', in which exploratory decisions are made in proportion to Bayesian uncertainty about whether other choices might produce outcomes that are better than the status quo. Quantitative model fits reveal that genetic factors modulate independent parameters of a reinforcement learning system.

Neurocomputational models of basal ganglia function in learning, memory and choice

The basal ganglia (BG) are critical for the coordination of several motor, cognitive, and emotional functions and become dysfunctional in several pathological states ranging from Parkinson's disease to Schizophrenia. Here we review principles developed within a neurocomputational framework of BG and related circuitry which provide insights into their functional roles in behavior. We focus on two classes of models: those that incorporate aspects of biological realism and constrained by functional principles, and more abstract mathematical models focusing on the higher level computational goals of the BG. While the former are arguably more "realistic", the latter have a complementary advantage in being able to describe functional principles of how the system works in a relatively simple set of equations, but are less suited to making specific hypotheses about the roles of specific nuclei and neurophysiological processes. We review the basic architecture and assumptions of these models, their relevance to our understanding of the neurobiological and cognitive functions of the BG, and provide an update on the potential roles of biological details not explicitly incorporated in existing models. Empirical studies ranging from those in transgenic mice to dopaminergic manipulation, deep brain stimulation, and genetics in humans largely support model predictions and provide the basis for further refinement. Finally, we discuss possible future directions and possible ways to integrate different types of models.