The role of signaling molecules in reward-related incentive learning

Fast heterosynaptic learning in a robot food retrieval task inspired by the limbic system

Hebbian learning is the most prominent paradigm in correlation based learning: if pre- and postsynaptic activity coincides the weight of the synapse is strengthened. Hebbian learning however, is not stable because of an autocorrelation term which causes the weights to grow exponentially. The standard solution would be to compensate the autocorrelation term. However, in this work we present a heterosynaptic learning rule which does not have an autocorrelation term and therefore does not show the instability of Hebbian learning. Consequently our heterosynaptic learning is much more stable than the classical Hebbian learning. The performance of our learning rule is demonstrated in a model which is inspired by the limbic system where an agent has to retrieve food.

Effort-related functions of nucleus accumbens dopamine and associated forebrain circuits

BACKGROUND

Over the last several years, it has become apparent that there are critical problems with the hypothesis that brain dopamine (DA) systems, particularly in the nucleus accumbens, directly mediate the rewarding or primary motivational characteristics of natural stimuli such as food. Hypotheses related to DA function are undergoing a substantial restructuring, such that the classic emphasis on hedonia and primary reward is giving way to diverse lines of research that focus on aspects of instrumental learning, reward prediction, incentive motivation, and behavioral activation.

OBJECTIVE

The present review discusses dopaminergic involvement in behavioral activation and, in particular, emphasizes the effort-related functions of nucleus accumbens DA and associated forebrain circuitry.

RESULTS

The effects of accumbens DA depletions on food-seeking behavior are critically dependent upon the work requirements of the task. Lever pressing schedules that have minimal work requirements are largely unaffected by accumbens DA depletions, whereas reinforcement schedules that have high work (e.g., ratio) requirements are substantially impaired by accumbens DA depletions. Moreover, interference with accumbens DA transmission exerts a powerful influence over effort-related decision making. Rats with accumbens DA depletions reallocate their instrumental behavior away from food-reinforced tasks that have high response requirements, and instead, these rats select a less-effortful type of food-seeking behavior.

CONCLUSIONS

Along with prefrontal cortex and the amygdala, nucleus accumbens is a component of the brain circuitry regulating effort-related functions. Studies of the brain systems regulating effort-based processes may have implications for understanding drug abuse, as well as energy-related disorders such as psychomotor slowing, fatigue, or anergia in depression.

Long-term upregulation of protein kinase A and adenylate cyclase levels in human smokers

Repeated injections of cocaine and morphine in laboratory rats cause a variety of molecular neuroadaptations in the cAMP signaling pathway in nucleus accumbens and ventral tegmental area. Here we report similar neuroadaptations in postmortem tissue from the brains of human smokers and former smokers. Activity levels of two major components of cAMP signaling, cAMP-dependent protein kinase A (PKA) and adenylate cyclase, were abnormally elevated in nucleus accumbens of smokers and in ventral midbrain dopaminergic region of both smokers and former smokers. Protein levels of the catalytic subunit of PKA were correspondingly higher in the ventral midbrain dopaminergic region of both smokers and former smokers. Protein levels of other candidate neuroadaptations, including glutamate receptor subunits, tyrosine hydroxylase, and other protein kinases, were within normal range. These findings extend our understanding of addiction-related neuroadaptations of cAMP signaling to tobacco smoking in human subjects and suggest that smoking-induced brain neuroadaptations can persist for significant periods in former smokers.

Dopamine and incentive learning: a framework for considering antipsychotic medication effects

Hyperfunction of brain dopamine (DA) systems is associated with psychosis in schizophrenia and the medications used to treat schizophrenia are DA receptor blockers. DA also plays a critical role in incentive learning produced by rewarding stimuli. Using DA as the link, these results suggest that psychosis in schizophrenia can be understood from the point of view of excessive incentive learning. Incentive learning is mediated through the non-declarative memory system and may rely on the striatum or medial prefrontal cortex depending on the task. Typical and atypical antipsychotics differentially affect expression of the immediate early gene c-fos, producing greater activity in the striatum and medial prefrontal cortex, respectively. This led to the hypothesis that performance of schizophrenic patients on tasks that depend on the striatum or medial prefrontal cortex will be differentially affected by their antipsychotic medication. Results from a number of published papers supported this dissociation. Furthermore, the effects of two atypical drugs, clozapine and olanzapine, on c-fos expression were different from another atypical, risperidone that resembles the typical antipsychotics. Similarly, in tests of incentive learning, risperidone acted like the typical antipsychotics. Thus, typical and atypical antipsychotic drugs differed in the types of cognitive performance they affected and, furthermore, members of the atypical class differed in their effects on cognition. It remains the task of researchers and clinicians to sort out the symptoms associated with the endogenous illness from possible iatrogenic symptoms resulting from the antipsychotic medications used to treat schizophrenia.

Nucleus accumbens PKA inhibition blocks acquisition but enhances expression of amphetamine-produced conditioned activity in rats

RATIONALE

The nucleus accumbens (NAc) plays a central role in dopamine-produced reward-related learning. In previous studies, the cyclic adenosine monophosphate-dependent protein kinase (PKA) inhibitor Rp-Cyclic 3',5'-hydrogen phosphorothioate adenosine triethylammonium salt (Rp-cAMPS) blocked the acquisition but not expression of NAc reward-related learning for natural rewards and the acquisition of psychostimulant drug conditioning.

OBJECTIVES

The current study assessed the role of PKA in the expression of NAc amphetamine (amph)-produced conditioning using conditioned activity (CA).

MATERIALS AND METHODS

After 5 days of habituation, a test environment was paired with bilateral NAc injections of amph (0.0 or 25.0 micro g) and the PKA inhibitor Rp-cAMPS (0.0, 5.0, 10.0, or 20.0 micro g) over three 60-min conditioning sessions separated by 48 h. To test for effects on expression, some groups received vehicle or amph alone before conditioning sessions and were injected with 0.0, 0.25, 5.0, or 20.0 mug of Rp-cAMPS before the single 60-min test session.

RESULTS

Amph produced acute increases in locomotion and robust CA. Rp-cAMPS impaired the acquisition of amph-produced CA but not its expression; in fact, it enhanced expression.

CONCLUSIONS

Results show that PKA inhibition blocks the acquisition but not the expression of amph-produced conditioning.

Intracerebral administration of protein kinase A or cAMP response element-binding protein antisense oligonucleotide can modulate amphetamine-mediated appetite suppression in free-moving rats

Although amphetamine (AMPH)-induced appetite suppression has been attributed to its inhibitory action on neuropeptide Y (NPY), an appetite neurotransmitter abundant in the brain, molecular mechanisms underlying this effect are not well known. This study examined the possible role of protein kinase A (PKA) and cAMP response element-binding protein (CREB) signaling in this anorectic effect, and the results showed that both PKA and CREB mRNA levels in hypothalamus were increased following AMPH treatment, which was relevant to a reduction of NPY mRNA level. To determine whether PKA or CREB was involved in the anorectic response, intracerebroventricular infusions of antisense oligonucleotide (or missense control) were performed 60 min before daily AMPH treatment in conscious rats, and results showed that either PKA or CREB knockdown could block AMPH-induced anorexia as well as restore NPY mRNA level, indicating the respective involvement of PKA and CREB signaling in the regulation of NPY gene expression. It is suggested that hypothalamic PKA and CREB signaling may involve the central regulation of AMPH-mediated feeding suppression via the modulation of NPY gene expression.

Intra-hippocampal inhibition of protein kinase AII attenuates morphine-induced conditioned place preference

Morphine and other drugs of abuse modulate protein kinase A (PKA) signaling within the mesolimbic reward pathway. Using a balanced conditioned place preference (CPP) paradigm, we studied the possible involvement of protein kinase AII (PKA II) on the acquisition, expression and consolidation of morphine place conditioning in male Wistar rats. Subcutaneous administration of various doses of morphine sulfate (1-9 mg/kg) induced CPP in a dose-dependent manner. H-89, a selective PKA II inhibitor, was administered into CA1 region of the hippocampus at 1, 2.5 and 5 microM/rat. Using a 3-day schedule of conditioning, it was found that the H-89 did not produce a significant place preference or place aversion. H-89 (1, 2.5 and 5 microM/rat) significantly reduced the time spent by rats in the morphine compartment when given immediately after each conditioning session (consolidation), whereas it had no effect when administered before morphine during the conditioning phase (acquisition) or before testing for place preference in the absence of morphine (expression). It is concluded that the PKA II may play an active role in the consolidation of reward-related memory of morphine in CA1 region of the hippocampus.

Neurotoxins and neurotoxicity mechanisms. an overview

Neurotoxins represent unique chemical tools, providing a means to 1) gain insight into cellular mechanisms of apopotosis and necrosis, 2) achieve a morphological template for studies otherwise unattainable, 3) specifically produce a singular phenotype of denervation, and 4) provide the starting point to delve into processes and mechanisms of nerve regeneration and sprouting. There are many other notable uses of neurotoxins in neuroscience research, and ever more being discovered each year. The objective of this review paper is to highlight the broad areas of neuroscience in which neurotoxins and neurotoxicity mechanism come into play. This shifts the focus away from neurotoxins per se, and onto the major problems under study today. Neurotoxins broadly defined are used to explore neurodegenerative disorders, psychiatric disorders and substance use disorders. Neurotoxic mechanisms relating to protein aggregates are indigenous to Alzheimer disease, Parkinson's disease. NeuroAIDS is a disorder in which microglia and macrophages have enormous import. The gap between the immune system and nervous system has been bridged, as neuroinflammation is now considered to be part of the neurodegenerative process. Related mechanisms now arise in the process of neurogenesis. Accordingly, the entire spectrum of neuroscience is within the purview of neurotoxins and neurotoxicity mechanisms. Highlights on discoveries in the areas noted, and on selective neurotoxins, are included, mainly from the past 2 to 3 years.

Strongly Improved Stability and Faster Convergence of Temporal Sequence Learning by Using Input Correlations Only

Currently all important, low-level, unsupervised network learning algorithms follow the paradigm of Hebb, where input and output activity are correlated to change the connection strength of a synapse. However, as a consequence, classical Hebbian learning always carries a potentially destabilizing autocorrelation term, which is due to the fact that every input is in a weighted form reflected in the neuron's output. This self-correlation can lead to positive feedback, where increasing weights will increase the output, and vice versa, which may result in divergence. This can be avoided by different strategies like weight normalization or weight saturation, which, however, can cause different problems. Consequently, in most cases, high learning rates cannot be used for Hebbian learning, leading to relatively slow convergence. Here we introduce a novel correlation-based learning rule that is related to our isotropic sequence order (ISO) learning rule (Porr & Wörgötter, 2003a), but replaces the derivative of the output in the learning rule with the derivative of the reflex input. Hence, the new rule uses input correlations only, effectively implementing strict heterosynaptic learning. This looks like a minor modification but leads to dramatically improved properties. Elimination of the output from the learning rule removes the unwanted, destabilizing autocorrelation term, allowing us to use high learning rates. As a consequence, we can mathematically show that the theoretical optimum of one-shot learning can be reached under ideal conditions with the new rule. This result is then tested against four different experimental setups, and we will show that in all of them, very few (and sometimes only one) learning experiences are needed to achieve the learning goal. As a consequence, the new learning rule is up to 100 times faster and in general more stable than ISO learning.

Behavioural Sensitization in Addiction, Schizophrenia, Parkinson’s Disease and Dyskinesia

Incentive learning takes place when dopaminergic neurons are activated, usually by rewards. As a result, previously neutral stimuli associated with reward acquire incentive salience and thus the ability to elicit approach or other responses in the future. Incentive learning is assumed to underlie psychostimulant-induced context-dependent sensitization that may play a prominent role in the development of addiction, in dyskinesia, and in amphetamine-induced psychosis. Assuming that these pathological states are due to the gradual process of sensitization, the effects of therapeutics might be manifested as a gradual desensitization. This assumption could explain the delay between onset of cellular effects of drugs (e.g., dopamine receptor blockade) and the improvement in symptoms (e.g., decreases in psychotic symptoms). Reduced dopamine activity results in behavioural changes that are opposite to psychostimulant-induced sensitization, i.e., rewarded behaviours decline in an extinction-like fashion despite the presence and consumption of rewards. We show here that also non reward-related behaviour, i.e., motor activity and catalepsy, follows the same rules: motility is not switched off by dopamine receptor blockade or by 6-hydroxydopamine lesions, but shows a test-to-test extinction-like decline. Thus, psychostimulant-induced sensitization and dopamine-deficiency induced decline of behaviour follows similar rules but in opposite directions.

Place preference induced by nucleus accumbens amphetamine is impaired by local blockade of Group II metabotropic glutamate receptors in rats

Background

The nucleus accumbens (NAc) plays a critical role in amphetamine-produced conditioned place preference (CPP). In previous studies, NAc basal and amphetamine-produced DA transmission was altered by Group II mGluR agents. We tested whether NAc amphetamine CPP depends on Group II mGluR transmission.

Results

NAc injections (0.5 μl/side) of the Group II mGluR antagonist (2 S)- a-ethylglutamic acid (EGLU: 0.01–0.8 μg but not 0.001 μg) impaired CPP. The drug did not block the acute locomotor effect of amphetamine.

Conclusion

Results suggest that Group II mGluRs may be necessary for the establishment of NAc amphetamine-produced CPP. These receptors may also mediate other forms of reward-related learning dependent on this structure.

Dopamine receptor stimulation modulates AMPA receptor synaptic insertion in prefrontal cortex neurons

Addiction is believed to involve glutamate-dependent forms of synaptic plasticity that promote the formation of new habits focused on drug seeking. We used primary cultures of rat prefrontal cortex (PFC) neurons to explore mechanisms by which dopamine-releasing psychomotor stimulants such as cocaine and amphetamine influence synaptic plasticity, focusing on AMPA receptor trafficking because of its key role in long-term potentiation (LTP). Brief stimulation of D1 dopamine receptors increased surface expression of glutamate receptor 1 (GluR1)-containing AMPA receptors through a protein kinase A-dependent mechanism, by increasing their rate of externalization at extrasynaptic sites. Newly externalized GluR1 remained extrasynaptic under basal conditions but could be translocated into synapses by subsequent NMDA receptor activation. These results suggest that D1 receptors may facilitate LTP by increasing the AMPA receptor pool available for synaptic insertion. However, stimulation of D2 receptors decreased surface and synaptic GluR1 expression. These findings are discussed in the context of evidence that D1 and D2 receptors act independently rather than antagonistically in the intact PFC. D1 receptor facilitation of AMPA receptor synaptic insertion helps explain D1 receptor-dependent facilitation of LTP and learning in the normal brain. Abnormal engagement of this mechanism during unregulated dopamine release may account for maladaptive plasticity after repeated exposure to cocaine or amphetamine.

Persistent changes in motivation to self-administer cocaine following modulation of cyclic AMP-dependent protein kinase A (PKA) activity in the nucleus accumbens

Drug-induced neuroadaptations within the nucleus accumbens, including activation of cAMP-dependent protein kinase A (PKA), may contribute to the synaptic plasticity and behavioural changes that underlie drug addiction. As a direct test of this hypothesis, we examined the effects in rats of PKA activation (Sp-cAMPS infusions of 10 and 20 nmol/side) and inhibition (Rp-cAMPS infusions of 10 and 20 nmol/side) in the nucleus accumbens on motivation to obtain cocaine as measured by responding under the progressive-ratio schedule. Bilateral infusions of Sp-cAMPS (20 nmol/side) resulted in an increase in progressive-ratio responding for cocaine and this effect persisted for several days. In contrast, Rp-cAMPS (20 nmol/side) produced persistent decreases in progressive-ratio responding for cocaine beginning on the day of administration and lasting for several days. These data suggest that alternations in PKA activity within the nucleus accumbens as a consequence of repeated cocaine exposure may contribute to addiction by producing persistent increases in motivation to obtain cocaine.

Differential effects of calcineurin inhibition and protein kinase A activation on nucleus accumbens amphetamine-produced conditioned place preference in rats

The nucleus accumbens (NAc) plays a critical role in amphetamine-produced conditioned place preference (CPP). In previous studies inhibition or activation of cyclic adenosine monophosphate-dependent protein kinase (PKA) blocked NAc amphetamine-produced CPP. PKA activation unrelated to ongoing DA transmission may disrupt reward-related learning. Calcineurin (CN) down-regulates downstream PKA targets. Unlike PKA activation, CN inhibition may preserve and enhance reward-related learning. The PKA signalling cascade is negatively regulated by calcineurin (CN). We tested the hypothesis that post-training CN inhibition in NAc will enhance NAc amphetamine-produced CPP and that PKA activation will block CPP. Eight but not four or two 30-min conditioning sessions were sufficient to establish significant CPP. Immediate post-training, NAc injection of the calcineurin inhibitor FK506 (5.0 but not 1.0 microg in 0.5 microL per side) led to a significant amphetamine CPP in rats receiving four but not two training sessions; the 5.0-microg dose had no effect on rats trained with eight sessions. Injections of the PKA activator Sp-cAMPS (2.5 or 10.0 microg in 0.5 microL per side) failed to affect CPP following two or four training sessions and blocked CPP produced by a standard 8-day conditioning schedule. Results suggest that CN acts as a negative regulator in the establishment of NAc amphetamine-produced CPP, a form of reward-related learning.

AMPA/kainate, NMDA, and dopamine D1 receptor function in the nucleus accumbens core: a context-limited role in the encoding and consolidation of instrumental memory

Neural integration of glutamate- and dopamine-coded signals within the nucleus accumbens (NAc) is a fundamental process governing cellular plasticity underlying reward-related learning. Intra-NAc core blockade of NMDA or D1 receptors in rats impairs instrumental learning (lever-pressing for sugar pellets), but it is not known during which phase of learning (acquisition or consolidation) these receptors are recruited, nor is it known what role AMPA/kainate receptors have in these processes. Here we show that pre-trial intra-NAc core administration of the NMDA, AMPA/KA, and D1 receptor antagonists AP-5 (1 microg/0.5 microL), LY293558 (0.01 or 0.1 microg/0.5 microL), and SCH23390 (1 microg/0.5 microL), respectively, impaired acquisition of a lever-pressing response, whereas post-trial administration left memory consolidation unaffected. An analysis of the microstructure of behavior while rats were under the influence of these drugs revealed that glutamatergic and dopaminergic signals contribute differentially to critical aspects of the initial, randomly emitted behaviors that enable reinforcement learning. Thus, glutamate and dopamine receptors are activated in a time-limited fashion-only being required while the animals are actively engaged in the learning context.

Instrumental learning, but not performance, requires dopamine D1-receptor activation in the amygdala

Substantial experimental evidence exists suggesting a critical role for dopamine in reinforcer-related processes, such as learning and drug addiction. Dopamine receptors, and in particular D1 receptors, are widely considered as modulators of synaptic plasticity. The amygdala contains both dopamine terminals and dopamine D1 receptors and is intimately involved in motivation and learning. However, little is known about the involvement of D1 receptor activation in two subnuclei of the mammalian amygdala, the central nucleus and basolateral complex in instrumental learning. Following recovery from surgery and preliminary training, rats with bilateral indwelling cannulae aimed at the central nucleus or basolateral complex were trained to lever-press for sucrose pellets over 12 sessions. Infusion of the selective D1 antagonist R(+)-7-chloro-8-hydroxy-3-methyl-1-phenyl-2,3,4,5-tetrahydro-1H-3-benzazepine hydrochloride (0.3 nmol and 3.0 nmol) prior to the first five training sessions dose-dependently impaired instrumental learning when compared with vehicle-infused controls. All rats were then exposed to five sessions drug-free; lever-pressing quickly reached equal levels across groups. A drug infusion prior to an 11th session revealed no effect on performance. Control experiments indicated that basic motivational processes and general motor responses were intact, such as spontaneous feeding and locomotor activity. These results show an essential role for D1-receptor activation in both the central nucleus and basolateral complex on the acquisition of lever pressing for sucrose pellets in rats, but not the performance of the behavior once conditioned. We propose that instrumental learning is dependent on plasticity in the central nucleus and basolateral complex amygdala, and that D1 receptor activation participates in transcriptional processes that underlie this plasticity.

Gene-environment interplay in schizopsychotic disorders

Genetic studies have sought to identify subtypes or endophenotypes of schizophrenia in an effort to improve the reliability of findings. A number of chromosomal regions or genes have now been shown to have had replicated linkage to schizophrenia susceptibility. Molecules involved in neurodevelopment or neurotransmitter function are coded by many of the genes that have been implicated in schizophrenia. Studies of neurotransmitter function have identified, among others, a possible role for GABA, glutamate and dopamine in animal models of schizophrenia. GABA neurons that co-express the calcium binding protein parvalbumin have been implicated as have glutamatergic metabotropic receptors and dopamine D3 receptors. Stress influences glutamate and dopamine providing another environmental factor that may interact with the influence of genes on neurotransmitter function. Neurotransmitter interactions include influences on signaling molecules and these too have been implicated in forms of learning thought to be affected in schizophrenia. Results continue to unravel the interplay of genes and environment in the etiology of schizophrenia and other psychotic disorders.