Effect of quinolinic acid-induced lesions of the subthalamic nucleus on performance on a progressive-ratio schedule of reinforcement: a quantitative analysis

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

Deep Brain Stimulation of the Subthalamic Nucleus Modulates Reward-Related Behavior: A Systematic Review

Deep brain stimulation of the subthalamic nucleus (STN-DBS) is an effective treatment for the motor symptoms of movement disorders including Parkinson's Disease (PD). Despite its therapeutic benefits, STN-DBS has been associated with adverse effects on mood and cognition. Specifically, apathy, which is defined as a loss of motivation, has been reported to emerge or to worsen following STN-DBS. However, it is often challenging to disentangle the effects of STN-DBS per se from concurrent reduction of dopamine replacement therapy, from underlying PD pathology or from disease progression. To this end, pre-clinical models allow for the dissociation of each of these factors, and to establish neural substrates underlying the emergence of motivational symptoms following STN-DBS. Here, we performed a systematic analysis of rodent studies assessing the effects of STN-DBS on reward seeking, reward motivation and reward consumption across a variety of behavioral paradigms. We find that STN-DBS decreases reward seeking in the majority of experiments, and we outline how design of the behavioral task and DBS parameters can influence experimental outcomes. While an early hypothesis posited that DBS acts as a "functional lesion," an analysis of lesions and inhibition of the STN revealed no consistent pattern on reward-related behavior. Thus, we discuss alternative mechanisms that could contribute to the amotivational effects of STN-DBS. We also argue that optogenetic-assisted circuit dissection could yield important insight into the effects of the STN on motivated behavior in health and disease. Understanding the mechanisms underlying the effects of STN-DBS on motivated behavior-will be critical for optimizing the clinical application of STN-DBS.

Neural substrates underlying effort, time, and risk-based decision making in motivated behavior

All mobile organisms rely on adaptive motivated behavior to overcome the challenges of living in an environment in which essential resources may be limited. A variety of influences ranging from an organism's environment, experiential history, and physiological state all influence a cost-benefit analysis which allows motivation to energize behavior and direct it toward specific goals. Here we review the substantial amount of research aimed at discovering the interconnected neural circuits which allow organisms to carry-out the cost-benefit computations which allow them to behave in adaptive ways. We specifically focus on how the brain deals with different types of costs, including effort requirements, delays to reward and payoff riskiness. An examination of this broad literature highlights the importance of the extended neural circuits which enable organisms to make decisions about these different types of costs. This involves Cortical Structures, including the Anterior Cingulate Cortex (ACC), the Orbital Frontal Cortex (OFC), the Infralimbic Cortex (IL), and prelimbic Cortex (PL), as well as the Baso-Lateral Amygdala (BLA), the Nucleus Accumbens (NAcc), the Ventral Pallidal (VP), the Sub Thalamic Nucleus (STN) among others. Some regions are involved in multiple aspects of cost-benefit computations while the involvement of other regions is restricted to information relating to specific types of costs.

Reduced Vglut2/Slc17a6 Gene Expression Levels throughout the Mouse Subthalamic Nucleus Cause Cell Loss and Structural Disorganization Followed by Increased Motor Activity and Decreased Sugar Consumption

The subthalamic nucleus (STN) plays a central role in motor, cognitive, and affective behavior. Deep brain stimulation (DBS) of the STN is the most common surgical intervention for advanced Parkinson's disease (PD), and STN has lately gained attention as target for DBS in neuropsychiatric disorders, including obsessive compulsive disorder, eating disorders, and addiction. Animal studies using STN-DBS, lesioning, or inactivation of STN neurons have been used extensively alongside clinical studies to unravel the structural organization, circuitry, and function of the STN. Recent studies in rodent STN models have exposed different roles for STN neurons in reward-related functions. We have previously shown that the majority of STN neurons express the vesicular glutamate transporter 2 gene (Vglut2/Slc17a6) and that reduction of Vglut2 mRNA levels within the STN of mice [conditional knockout (cKO)] causes reduced postsynaptic activity and behavioral hyperlocomotion. The cKO mice showed less interest in fatty rewards, which motivated analysis of reward-response. The current results demonstrate decreased sugar consumption and strong rearing behavior, whereas biochemical analyses show altered dopaminergic and peptidergic activity in the striatum. The behavioral alterations were in fact correlated with opposite effects in the dorsal versus the ventral striatum. Significant cell loss and disorganization of the STN structure was identified, which likely accounts for the observed alterations. Rare genetic variants of the human VGLUT2 gene exist, and this study shows that reduced Vglut2/Slc17a6 gene expression levels exclusively within the STN of mice is sufficient to cause strong modifications in both the STN and the mesostriatal dopamine system.

Evaluation of a Postoperative Pain-Like State on Motivated Behavior in Rats: Effects of Plantar Incision on Progressive-Ratio Food-Maintained Responding

There has been recent interest in characterizing the effects of pain-like states on motivated behaviors in order to quantify how pain modulates goal-directed behavior and the persistence of that behavior. The current set of experiments assessed the effects of an incisional postoperative pain manipulation on food-maintained responding under a progressive-ratio (PR) operant schedule. Independent variables included injury state (plantar incision or anesthesia control) and reinforcer type (grain pellet or sugar pellet); dependent variables were tactile sensory thresholds and response breakpoint. Once responding stabilized on the PR schedule, separate groups of rats received a single ventral hind paw incision or anesthesia (control condition). Incision significantly reduced breakpoints in rats responding for grain, but not sugar. In rats responding for sugar, tactile hypersensitivity recovered within 24 hr, indicating a faster recovery of incision-induced tactile hypersensitivity compared to rats responding for grain, which demonstrated recovery at PD2. The NSAID analgesic, diclofenac (5.6 mg/kg) completely restored incision-depressed PR operant responding and tactile sensitivity at 3 hr following incision. The PR schedule differentiated between sucrose and grain, suggesting that relative reinforcing efficacy may be an important determinant in detecting pain-induced changes in motivated behavior.

Quantitative analysis of performance on a progressive-ratio schedule: effects of reinforcer type, food deprivation and acute treatment with Δ⁹-tetrahydrocannabinol (THC)

Rats' performance on a progressive-ratio schedule maintained by sucrose (0.6M, 50 μl) and corn oil (100%, 25 μl) reinforcers was assessed using a model derived from Killeen's (1994) theory of schedule-controlled behaviour, 'Mathematical Principles of Reinforcement'. When the rats were maintained at 80% of their free-feeding body weights, the parameter expressing incentive value, a, was greater for the corn oil than for the sucrose reinforcer; the response-time parameter, δ, did not differ between the reinforcer types, but a parameter derived from the linear waiting principle (T0), indicated that the minimum post-reinforcement pause was longer for corn oil than for sucrose. When the rats were maintained under free-feeding conditions, a was reduced, indicating a reduction of incentive value, but δ was unaltered. Under the food-deprived condition, the CB1 cannabinoid receptor agonist Δ(9)-tetrahydrocannabinol (THC: 0.3, 1 and 3 mg kg(-1)) increased the value of a for sucrose but not for corn oil, suggesting a selective enhancement of the incentive value of sucrose; none of the other parameters was affected by THC. The results provide new information about the sensitivity of the model's parameters to deprivation and reinforcer quality, and suggest that THC selectively enhances the incentive value of sucrose.

Neuronal activity correlated with checking behaviour in the subthalamic nucleus of patients with obsessive-compulsive disorder

Doubt, and its behavioural correlate, checking, is a normal phenomenon of human cognition that is dramatically exacerbated in obsessive-compulsive disorder. We recently showed that deep brain stimulation in the associative-limbic area of the subthalamic nucleus, a central core of the basal ganglia, improved obsessive-compulsive disorder. To understand the physiological bases of symptoms in such patients, we recorded the activity of individual neurons in the therapeutic target during surgery while subjects performed a cognitive task that gave them the possibility of unrestricted repetitive checking after they had made a choice. We postulated that the activity of neurons in this region could be influenced by doubt and checking behaviour. Among the 63/87 task-related neurons recorded in 10 patients, 60% responded to various combinations of instructions, delay, movement or feedback, thus highlighting their role in the integration of different types of information. In addition, task-related activity directed towards decision-making increased during trials with checking in comparison with those without checking. These results suggest that the associative-limbic subthalamic nucleus plays a role in doubt-related repetitive thoughts. Overall, our results not only provide new insight into the role of the subthalamic nucleus in human cognition but also support the fact that subthalamic nucleus modulation by deep brain stimulation reduced compulsive behaviour in patients with obsessive-compulsive disorder.

Pathological gambling in Parkinson's disease: subthalamic oscillations during economics decisions

Pathological gambling develops in up to 8% of patients with Parkinson's disease. Although the pathophysiology of gambling remains unclear, several findings argue for a dysfunction in the basal ganglia circuits. To clarify the role of the subthalamic nucleus in pathological gambling, we studied its activity during economics decisions. We analyzed local field potentials recorded from deep brain stimulation electrodes in the subthalamic nucleus while parkinsonian patients with (n = 8) and without (n = 9) pathological gambling engaged in an economics decision-making task comprising conflictual trials (involving possible risk-taking) and non conflictual trials. In all parkinsonian patients, subthalamic low frequencies (2-12 Hz) increased during economics decisions. Whereas, in patients without gambling, low-frequency oscillations exhibited a similar pattern during conflictual and non conflictual stimuli, in those with gambling, low-frequency activity increased significantly more during conflictual than during non conflictual stimuli. The specific low-frequency oscillatory pattern recorded in patients with Parkinson's disease who gamble could reflect a subthalamic dysfunction that makes their decisional threshold highly sensitive to risky options. When parkinsonian patients process stimuli related to an economics task, low-frequency subthalamic activity increases. This task-related change suggests that the cognitive-affective system that drives economics decisional processes includes the subthalamic nucleus. The specific subthalamic neuronal activity during conflictual decisions in patients with pathological gambling supports the idea that the subthalamic nucleus is involved in behavioral strategies and in the pathophysiology of gambling.

Effects of amisulpride and aripiprazole on progressive-ratio schedule performance: comparison with clozapine and haloperidol

Clozapine and some other atypical antipsychotics (e.g. quetiapine, olanzapine) have been found to exert a characteristic profile of action on operant behaviour maintained by progressive-ratio schedules, as revealed by Killeen's Mathematical Principles of Reinforcement model of schedule-controlled behaviour. These drugs increase the value of a parameter that expresses the 'incentive value' of the reinforcer (a) and a parameter that is inversely related to the organism's 'motor capacity' (δ). This experiment examined the effects of two further atypical antipsychotics, aripiprazole and amisulpride, on progressive-ratio schedule performance in rats; the effects of clozapine and a conventional antipsychotic, haloperidol, were also examined. In agreement with previous findings, clozapine (4, 8 mg kg⁻¹) increased a and δ, whereas haloperidol (0.05, 0.1 mg kg⁻¹) reduced a and increased δ. Aripiprazole (3,30 mg kg⁻¹) increased δ but did not affect a. Amisulpride (5, 50 mg kg⁻¹) had a delayed and protracted effect: δ was increased 3-6 hours after treatment; a was increased 1.5 hours, and reduced 12-24 hours after treatment. Interpretation based on Killeen's model suggests that aripiprazole does not share clozapine's ability to enhance reinforcer value. Amisulpride produced a short-lived enhancement, followed by a long-lasting reduction, of reinforcer value. Both drugs impaired motor performance.

A clozapine-like effect of cyproheptadine on progressive ratio schedule performance

The atypical antipsychotic drug clozapine has multiple pharmacological actions, some of which, including 5-hydroxytryptamine (5-HT₂) and histamine (H₁) receptor antagonist effects, are shared by the non-selective 5-HT receptor antagonist cyproheptadine. Atypical antipsychotics have a characteristic profile of action on operant behaviour maintained by progressive ratio schedules, as revealed by Killeen's (1994) mathematical model of schedule controlled behaviour. These drugs increase the values of a parameter that expresses the 'incentive value' of the reinforcer (a) and a parameter that is inversely related to the 'motor capacity' of the organism (δ). This experiment examined the effects of acute treatment with cyproheptadine and clozapine on performance on a progressive ratio schedule of food reinforcement in rats; the effects of a conventional antipsychotic, haloperidol, and two drugs with food intake-enhancing effects, chlordiazepoxide and Δ⁹-tetrahydrocannabinol (THC), were also examined. Cyproheptadine (1, 5 mg kg⁻¹) and clozapine (3.75, 7.5 mg kg⁻¹) increased a and δ. Haloperidol (0.05, 0.1 mg kg⁻¹) reduced a and increased δ. Chlordiazepoxide (3, 10 mg kg⁻¹) increased a but reduced δ. THC (1, 3 mg kg⁻¹) had no effect. Interpretation based on Killeen's (1994) model suggests that cyproheptadine and clozapine enhanced the incentive value of the reinforcer and impaired motor performance. Motor impairment may be due to sedation (possibly reflecting H₁ receptor blockade). Enhancement of incentive value may reflect simultaneous blockade of H₁ and 5-HT₂ receptors, which has been proposed as the mechanism underlying the food intake-enhancing effect of cyproheptadine. In agreement with previous findings, haloperidol impaired motor performance and reduced the incentive value of the reinforcer. Chlordiazepoxide's effect on a is consistent with its food intake-enhancing effect.

Effect of orexin-B-saporin-induced lesions of the lateral hypothalamus on performance on a progressive ratio schedule

It has been suggested that a sub-population of orexinergic neurones whose somata lie in the lateral hypothalamic area (LHA) play an important role in regulating the reinforcing value of both food and drugs. This experiment examined the effect of disruption of orexinergic mechanisms in the LHA on performance on the progressive ratio schedule of reinforcement, in which the response requirement increases progressively for successive reinforcers. The data were analysed using a mathematical model which yields a quantitative index of reinforcer value and dissociates effects of interventions on motor and motivational processes. Rats were trained under a progressive ratio schedule using food-pellet reinforcement. They received bilateral injections of conjugated orexin-B-saporin (OxSap) into the LHA or sham lesions. Training continued for a further 40 sessions after surgery. Equations were fitted to the response rate data from each rat, and the parameters of the model were derived for successive blocks of 10 sessions. The OxSap lesion reduced the number of orexin-containing neurones in the LHA by approximately 50% compared with the sham-lesioned group. The parameter expressing the incentive value of the reinforcer was not significantly altered by the lesion. However, the parameter related to the maximum response rate was significantly affected, suggesting that motor capacity was diminished in the OxSap-lesioned group. The results indicate that OxSap lesions of the LHA disrupted food-reinforced responding on the progressive ratio schedule. It is suggested that this disruption was brought about by a change in non-motivational (motor) processes.

An examination of the effects of subthalamic nucleus inhibition or μ-opioid receptor stimulation on food-directed motivation in the non-deprived rat

The subthalamic nucleus (STN) serves important functions in regulating movement, cognition, and motivation and is connected with cortical and basal ganglia circuits that process reward and reinforcement. In order to further examine the role of the STN on motivation toward food in non-deprived rats, these experiments studied the effects of pharmacological inhibition or μ-opioid receptor stimulation of the STN on the 2-h intake of a sweetened fat diet, the amount of work exerted to earn sucrose on a progressive ratio 2 (PR-2) schedule of reinforcement, and performance on a differential reinforcement of low-rate responding (DRL) schedule for sucrose reward. Separate behavioral groups (N=6-9) were tested following bilateral inhibition of the STN with the GABA(A) receptor agonist muscimol (at 0-5 ng/0.5 μl/side) or following μ-opioid receptor stimulation with the agonist D-Ala², N-MePhe⁴, Gly-ol-enkephalin (DAMGO; at 0, 0.025 or 0.25 μg/0.5 μl/side). Although STN inhibition increased ambulatory behavior during 2-h feeding sessions, it did not significantly alter intake of the sweetened fat diet. STN inhibition also did not affect the breakpoint for sucrose pellets during a 1-h PR-2 reinforcement schedule or impact the number of reinforcers earned on a 1-h DRL-20s reinforcement schedule in non-deprived rats. In contrast, STN μ-opioid receptor stimulation significantly increased feeding on the palatable diet and reduced the reinforcers earned on a DRL-20 schedule, although DAMGO microinfusions had no effect on PR-2 performance. These data suggest that STN inhibition does not enhance incentive motivation for food in the absence of food restriction and that STN μ-opioid receptors play an important and unique role in motivational processes.

Quantitative analysis of the effect of lesions of the subthalamic nucleus on intertemporal choice: further evidence for enhancement of the incentive value of food reinforcers

Recent evidence suggests that the subthalamic nucleus (STN) is involved in regulating the incentive value of food reinforcers. The objective of this study was to examine the effect of lesions of the STN on intertemporal choice (choice between reinforcers differing in size and delay). Rats with bilateral quinolinic acid-induced lesions of the STN (n = 15) or sham lesions (n = 14) were trained in a discrete-trials progressive delay schedule to press levers A and B for a sucrose solution. Responses on A delivered 50 microl of the solution after a delay d(A); responses on B delivered 100 microl after a delay d(B). d(B) increased across blocks of trials; d(A) was manipulated across phases of the experiment. Indifference delay, d(B(50)) (value of d(B) corresponding to 50% choice of B), was estimated for each rat in each phase, and linear indifference functions (d(B(50)) vs. d(A)) were derived. The STN-lesioned group showed a flatter slope of the indifference function (implying higher instantaneous reinforcer values) than the sham-lesioned group; the intercepts did not differ between the groups. The results agree with recent evidence for a role of the STN in incentive value. Unlike some earlier studies, these results do not indicate a role of the STN in delay discounting.

Effects of chronic administration of drugs of abuse on impulsive choice (delay discounting) in animal models

Drug-addicted individuals show high levels of impulsive choice, characterized by preference for small immediate over larger but delayed rewards. Although the causal relationship between chronic drug use and elevated impulsive choice in humans has been unclear, a small but growing body of literature over the past decade has shown that chronic drug administration in animal models can cause increases in impulsive choice, suggesting that a similar causal relationship may exist in human drug users. This article reviews this literature, with a particular focus on the effects of chronic cocaine administration, which have been most thoroughly characterized. The potential mechanisms of these effects are described in terms of drug-induced neural alterations in ventral striatal and prefrontal cortical brain systems. Some implications of this research for pharmacological treatment of drug-induced increases in impulsive choice are discussed, along with suggestions for future research in this area.