MPR

Effect of reinforcer magnitude on performance maintained by progressive-ratio schedules

This experiment examined the relationship between reinforcer magnitude and quantitative measures of performance on progressive-ratio schedules. Fifteen rats were trained under a progressive-ratio schedule in seven phases of the experiment in which the volume of a 0.6-M sucrose solution reinforcer was varied within the range 6-300 microl. Overall response rates in successive ratios conformed to a bitonic equation derived from Killeen's (1994) Mathematical Principles of Reinforcement. The "specific activation" parameter, a, which is presumed to reflect the incentive value of the reinforcer, was a monotonically increasing function of reinforcer volume; the "response time" parameter, delta, which defines the minimum response time, increased as a function of reinforcer volume; the "currency" parameter, beta, which is presumed to reflect the coupling of responses to the reinforcer, declined as a function of volume. Running response rate (response rate calculated after exclusion of the postreinforcement pause) decayed monotonically as a function of ratio size; the index of curvature of this function increased as a function of reinforcer volume. Postreinforcement pause increased as a function of ratio size. Estimates of a derived from overall response rates and postreinforcement pauses showed a modest positive correlation across conditions and between animals. Implications of the results for the quantification of reinforcer value and for the use of progressive-ratio schedules in behavioral neuroscience are discussed.

Effect of quinolinic acid-induced lesions of the subthalamic nucleus on performance on a progressive-ratio schedule of reinforcement: a quantitative analysis

The subthalamic nucleus (STN), a major relay in the indirect striatofugal pathway, plays an important role in extrapyramidal motor control. Recent evidence indicates that it may also be involved in regulating the incentive value of food reinforcers.

OBJECTIVE

To examine the effect of lesions of the STN on performance on a progressive-ratio schedule using a quantitative model that dissociates effects of interventions on motor and motivational processes [Killeen PR. Mathematical principles of reinforcement. Behav Brain Sci 1994;17:105-72]. Rats with bilateral quinolinic acid-induced lesions of the STN (n=14) or sham lesions (n=14) were trained to press a lever for food-pellet reinforcers under a progressive-ratio schedule. In Phase 1 (90 sessions) the reinforcer was one pellet; in Phase 2 (30 sessions) it was two pellets; in Phase 3 (30 sessions) it was again one pellet.

RESULTS

The performance of both groups conformed to the model of progressive-ratio schedule performance. The motor parameter, delta, was significantly higher in the STN-lesioned than the sham-lesioned group, reflecting lower overall response rates in the lesioned group. The motivational parameter, a, was significantly higher in the STN-lesioned group than in the sham-lesioned group, consistent with enhanced reinforcer value in the STN-lesioned group compared to the sham-lesioned group. In both groups, a was sensitive to changes in reinforcer size, being significantly greater under the two-pellet condition (Phase 2) than under the one-pellet condition (Phases 1 and 3). The results suggest that destruction of the STN impairs response capacity and enhances the incentive value of food reinforcers.

Modeling operant behavior in the Parkinsonian rat

Mathematical principles of reinforcement (MPR; Killeen, 1994) is a quantitative model of operant behavior that contains three parameters representing motor capacity (delta), motivation (a), and short term memory (lambda). The present study applied MPR to characterize the effects of bilateral infusions of 6-OHDA into the substantia nigra pars compacta in the rat, a model of Parkinson's disease. Rats were trained to lever press under a 5-component fixed-ratio (5, 15, 30, 60, and 100) schedule of food reinforcement. Rats were tested for 15 days prior to dopamine lesions and again for 15 days post-lesion. To characterize functional loss relative to lesion size, rats were grouped according to the extent and the degree of lateralization of their dopamine loss. Response rates decreased as a function of dopamine depletion, primarily at intermediate ratios. MPR accounted for 98% of variance in pre- and post-lesion response rates. Consistent with reported disruptions in motor behavior induced by dopaminergic lesions, estimates of delta increased when dopamine was severely depleted. There was no support for different estimates of a based on pre- and post-lesion performance of any lesion group, suggesting that dopamine loss has negligible effects on incentive motivation. The present study demonstrates the usefulness of combining operant techniques with a theoretical model to better understand the effects of a neurochemical manipulation.

Research review: dopamine transfer deficit: a neurobiological theory of altered reinforcement mechanisms in ADHD

This review considers the hypothesis that changes in dopamine signalling might account for altered sensitivity to positive reinforcement in children with ADHD. The existing evidence regarding dopamine cell activity in relation to positive reinforcement is reviewed. We focus on the anticipatory firing of dopamine cells brought about by a transfer of dopamine cell responses to cues that precede reinforcers. It is proposed that in children with ADHD there is diminished anticipatory dopamine cell firing, which we call the dopamine transfer deficit (DTD). The DTD theory leads to specific and testable predictions for human and animal research. The extent to which DTD explains symptoms of ADHD and effects of pharmacological interventions is discussed. We conclude by considering the neural changes underlying the etiology of DTD.

Effect of quinolinic acid-induced lesions of the nucleus accumbens core on performance on a progressive ratio schedule of reinforcement: implications for inter-temporal choice

RATIONALE

The nucleus accumbens core (AcbC) is believed to contribute to the control of operant behaviour by reinforcers. Recent evidence suggests that it is not crucial for determining the incentive value of immediately available reinforcers, but is important for maintaining the values of delayed reinforcers.

OBJECTIVE

This study aims to examine the effect of AcbC lesions on performance on a progressive-ratio schedule using a quantitative model that dissociates effects of interventions on motor and motivational processes (Killeen 1994 Mathematical principles of reinforcement. Behav Brain Sci 17:105-172).

MATERIALS AND METHODS

Rats with bilateral quinolinic acid-induced lesions of the AcbC (n = 15) or sham lesions (n = 14) were trained to lever-press for food-pellet reinforcers under a progressive-ratio schedule. In Phase 1 (90 sessions) the reinforcer was one pellet; in Phase 2 (30 sessions), it was two pellets; in Phase 3, (30 sessions) it was one pellet.

RESULTS

The performance of both groups conformed to the model of progressive-ratio performance (group mean data: r2 > 0.92). The motor parameter, delta, was significantly higher in the AcbC-lesioned than the sham-lesioned group, reflecting lower overall response rates in the lesioned group. The motivational parameter, a, was sensitive to changes in reinforcer size, but did not differ significantly between the two groups. The AcbC-lesioned group showed longer post-reinforcement pauses and lower running response rates than the sham-lesioned group.

CONCLUSIONS

The results suggest that destruction of the AcbC impairs response capacity but does not alter the efficacy of food reinforcers. The results are consistent with recent findings that AcbC lesions do not alter sensitivity to reinforcer size in inter-temporal choice schedules.

Behavioral variability, elimination of responses, and delay-of-reinforcement gradients in SHR and WKY rats

Background

Attention-deficit/hyperactivity disorder (ADHD) is characterized by a pattern of inattention, hyperactivity, and impulsivity that is cross-situational, persistent, and produces social and academic impairment. Research has shown that reinforcement processes are altered in ADHD. The dynamic developmental theory has suggested that a steepened delay-of-reinforcement gradient and deficient extinction of behavior produce behavioral symptoms of ADHD and increased behavioral variability.

Method

The present study investigated behavioral variability and elimination of non-target responses during acquisition in an animal model of ADHD, the spontaneously hypertensive rat (SHR), using Wistar Kyoto (WKY) rats as controls. The study also aimed at providing a novel approach to measuring delay-of-reinforcement gradients in the SHR and the WKY strains. The animals were tested in a modified operant chamber presenting 20 response alternatives. Nose pokes in a target hole produced water according to fixed interval (FI) schedules of reinforcement, while nose pokes in the remaining 19 holes either had no consequences or produced a sound or a short flickering of the houselight. The stimulus-producing holes were included to test whether light and sound act as sensory reinforcers in SHR.

Data from the first six sessions testing FI 1 s were used for calculation of the initial distribution of responses. Additionally, Euclidean distance (measured from the center of each hole to the center of the target hole) and entropy (a measure of variability) were also calculated.

Delay-of-reinforcement gradients were calculated across sessions by dividing the fixed interval into epochs and determining how much reinforcement of responses in one epoch contributed to responding in the next interval.

Results

Over the initial six sessions, behavior became clustered around the target hole. There was greater initial variability in SHR behavior, and slower elimination of inefficient responses compared to the WKY. There was little or no differential use of the stimulus-producing holes by either strain. For SHR, the reach of reinforcement (the delay-of-reinforcement gradient) was restricted to the preceding one second, whereas for WKY it extended about four times as far.

Conclusion

The present findings support previous studies showing increased behavioral variability in SHR relative to WKY controls. A possibly related phenomenon may be the slowed elimination of non-operant nose pokes in SHR observed in the present study. The findings provide support for a steepened delay-of-reinforcement gradient in SHR as suggested in the dynamic developmental theory of ADHD. Altered reinforcement processes characterized by a steeper and shorter delay-of-reinforcement gradient may define an ADHD endophenotype.

Reaction times in discriminations of varying difficulty: Decision modulated by arousal

Pigeons discriminated the hue of a spot of light that appeared in discrete trials. A green spot always signaled food ("green S+") and so did a red spot of constant hue ("red S+"), but on most trials a different red hue appeared and no food was given for pecks ("red S-"). The hue of red S- stayed the same during blocks of up to twelve sessions, but it changed from block to block. During a final group of sessions red S- was omitted and the percent reinforcement to the two S+ stimuli was varied. Major findings were that (1) percent response (Pct(R)) to S- varied with S+/S- similarity, describing a typical ogival psychometric function; (2) Reaction times (RTs) to both red and green S+s were minimal when the red discrimination was impossible (that is, when red S+ and S- were identical); (3) RTs to red S+ were greatest during discriminations of intermediate difficulty; (4) as Pct(R) declined during the learning of a difficult discrimination, RTs increased to red S+ as well as red S-. Most aspects of the data were reproduced by simulations with a quantitative model that incorporated reinforcement-based decision and arousal processes.

Chronic stress impairs spatial memory and motivation for reward without disrupting motor ability and motivation to explore

This study uses an operant, behavioral model to assess the daily changes in the decay rate of short-term memory, motivation, and motor ability in rats exposed to chronic restraint. Restraint decreased reward-related motivation by 50% without altering memory decay rate or motor ability. Moreover, chronic restraint impaired hippocampal-dependent spatial memory on the Y maze (4-hr delay) and produced CA3 dendritic retraction without altering hippocampal-independent maze navigation (1-min delay) or locomotion. Thus, mechanisms underlying motivation for food reward differ from those underlying Y maze exploration, and neurobiological substrates of spatial memory, such as the hippocampus, differ from those that underlie short-term memory. Chronic restraint produces functional, neuromorphological, and physiological alterations that parallel symptoms of depression in humans.

Operant variability when reinforcement is delayed

Operant responses are often weakened when delays are imposed between the responses and reinforcers. We examined what happens when delayed reinforcers were contingent upon operant response variability. Three groups of rats were rewardedfor varying their response sequences, with onegroup rewarded for high variability, another for middle, and the third for low levels. Consistent with many reports in the literature, responding slowed significantly in all groups as delays were lengthened. Consistent with other reports, large differences in variability were maintained across the three groups despite the delays. Reinforced variability appears to be relatively immune to disruption by such things as delays, response slowing, prefeeding, and noncontingent reinforcement. Furthermore, the small effects on variability depended on baseline levels: As delays lengthened, variability increased in the low group, was statistically unchanged in the middle group, and decreased in the high group, an interaction similar to that reported previously when reinforcement frequencies were lowered. Thus, variable operant responding is controlled by reinforcement contingencies, but sometimes differently than more commonly studied repetitive responding.

Does satiation close the open economy?

Pigeons responded on fixed-interval and fixed-ratio food schedules during sessions of extended duration. Pause lengths from the beginning of the session, when the subjects were hungry, resembled those found in open economies, whereas pause lengths from the end of the sessions, when the subjects were close to satiation, resembled those from closed economies. A model of motivation captured key features of the data, suggesting that a changing level of hunger is a causal factor in the behavioral differences observed between open and closed economies. Behavioral theories may provide a parsimonious alternative to economic theories in accounting for such effects.

The effects of interval duration on temporal tracking and alternation learning

On cyclic-interval reinforcement schedules, animals typically show a postreinforcement pause that is a function of the immediately preceding time interval (temporal tracking). Animals, however, do not track single-alternation schedules-when two different intervals are presented in strict alternation on successive trials. In this experiment, pigeons were first trained with a cyclic schedule consisting of alternating blocks of 12 short intervals (5 s or 30 s) and 12 long intervals (180 s), followed by three different single-alternation interval schedules: (a) 30 s and 180 s, (b) 5 s and 180 s, and (c) 5 s and 30 s. Pigeons tracked both schedules with alternating blocks of 12 intervals. With the single-alternation schedules, when the short interval duration was 5 s, regardless of the duration of the longer interval, pigeons learned the alternation pattern, and their pause anticipated the upcoming interval. When the shorter interval was 30 s, even when the ratio of short to long intervals was kept at 6:1, pigeons did not initially show anticipatory pausing-a violation of the principle of timescale invariance.

Comparison of the effects of clozapine and 8-hydroxy-2-(di-n-propylamino)tetralin (8-OH-DPAT) on progressive ratio schedule performance: evidence against the involvement of 5-HT1A receptors in the behavioural effects of clozapine

RATIONALE

Performance on progressive ratio schedules has been proposed as a means of assessing the effects of drugs on the efficacy of reinforcers. A mathematical model (Killeen PR (1994) Mathematical principles of reinforcement. Behav Brain Sci 17:105-172) affords a basis for quantifying the effects of drugs on progressive ratio schedule performance. The model postulates a bitonic function relating response rate and ratio size. One parameter of the function, a, expresses the motivational effect of the reinforcer, whereas another parameter, delta, expresses the minimum time needed to execute a response, and is regarded as an index of 'motor capacity'. Previously we found that the atypical antipsychotic clozapine increased a, indicating an increase in reinforcer efficacy; a similar effect was observed with the 5-hydroxytryptamine (5-HT)(1A) receptor agonist 8-hydroxy-2-(di-n-propylamino)tetralin (8-OH-DPAT). It has been suggested that some of clozapine's behavioural effects are mediated by agonistic action at 5-HT(1A) receptors.

OBJECTIVE

This study was conducted to compare the effects of clozapine and 8-OH-DPAT on progressive ratio schedule performance.

METHODS

Rats were trained under a time-constrained progressive ratio schedule (50-min sessions). In experiment 1, they received acute doses of clozapine (4 mg kg(-1)) and 8-OH-DPAT (100 microg kg(-1)), alone and in combination with the 5-HT(1A) receptor antagonist N-[2-(4-[2-methoxyphenyl]-1-piperazinyl)ethyl]-N-2-yridinylcyclohexanecarboxamide (WAY-100635; 30 microg kg(-1)). In experiment 2, the effects of clozapine (2, 4 and 8 mg kg(-1)) and 8-OH-DPAT (25, 50 and 100 microg kg(-1)) were compared between intact rats and rats whose 5-HTergic pathways had been ablated by 5,7-dihydroxytryptamine (5,7-DHT).

RESULTS

In both experiments, clozapine and 8-OH-DPAT increased a and delta. In experiment 1, WAY-100635 abolished the effect of 8-OH-DPAT on a and delta, but did not alter clozapine's effects on these parameters. In experiment 2, the effects of clozapine and 8-OH-DPAT did not differ between sham-lesioned and 5,7-DHT-lesioned rats.

CONCLUSIONS

The results confirm previous findings on the effects of clozapine and 8-OH-DPAT on progressive ratio schedule performance. 8-OH-DPAT's effects are probably mediated by post-synaptic 5-HT(1A) receptors; clozapine's effects are mediated by a different mechanism, which does not appear to involve 5-HT(1A) receptors and which does not depend upon an intact 5-HTergic pathway.

Quantitative analysis of the effects of some "atypical" and "conventional" antipsychotics on progressive ratio schedule performance

RATIONALE

Performance on progressive ratio schedules has been proposed as a means of assessing the effects of drugs on the value or "efficacy" of reinforcers. A mathematical model affords a basis for quantifying the effects of drugs on progressive ratio schedule performance. According to this model, the relation between response rate and ratio size is described by a bitonic (inverted-U) function. One parameter of the function, alpha, expresses the motivational or "activating" effect of the reinforcer (duration of activation of responding produced by the reinforcer), whereas another parameter, delta, expresses the minimum time needed to execute a response, and is regarded as an index of "motor capacity". In a previous experiment we found that the "atypical" antipsychotic clozapine increased alpha, indicating an increase in the efficacy of a food reinforcer.

OBJECTIVE

We examined the effects of four "atypical" and four "conventional" antipsychotics on progressive ratio schedule performance.

METHODS

Rats responded for a sucrose reinforcer (0.6 M, 50 microl) on a time-constrained progressive ratio schedule (50-min sessions). After 90 preliminary training sessions, they received acute doses of antipsychotics (doses in mg kg(-1)): atypical: clozapine (2, 4, 8, IP; n=15), quetiapine (1.25, 2.5, 5, 10, SC; n=23), olanzapine (0.25, 0.5, 1, IP; n=15), ziprasidone (0.625, 1.25, 2.5, IP, n=15); conventional: haloperidol (0.025, 0.05, 0.1, IP, n=15), pimozide (0.125, 0.25, 0.5, IP; n=15), raclopride (0.25, 0.5, 1, SC; n=12), cis-flupenthixol (0.2, 0.4, 0.8, SC; n=15). Values of a and delta were estimated from the response rate functions obtained under each treatment condition, and were compared between drug and vehicle-alone treatments.

RESULTS

The atypical antipsychotics significantly increased alpha (indicating enhancement of reinforcer efficacy), and also increased delta (indicating reduction of motor capacity). Haloperidol, pimozide and raclopride significantly increased delta; none of the conventional antipsychotics significantly altered alpha.

CONCLUSIONS

The results extend previous findings with clozapine to other atypical antipsychotics and suggest that enhancement of the efficacy of reinforcers may be a common feature of atypical antipsychotics not shared by conventional antipsychotics.

Social interaction opportunity and intermittent presentations of ethanol sipper tube induce ethanol drinking in rats

We evaluated the effects of social interaction opportunity (SIO) and intermittent presentations of the ethanol sipper tube (IS) on autoshaping of ethanol drinking in nondeprived rats. Rats were assigned to one of seven groups. Two groups experienced brief IS, either paired with or randomly related to the response-independent raising of a guillotine door (D) revealing the presence of a conspecific male rat in a holding cage (SIO). Two control groups received similar training, respectively, except that the D revealed an empty cage, whereas a third control group received IS but neither D nor SIO. For two additional control groups, the ethanol sipper tube was continuously available during the session, with and without SIO, with both groups receiving intermittent D. In IS conditions, procedures with SIO induced more ethanol intake than did non-SIO procedures, indicating that SIO contributed to ethanol intake, but D procedures did not differ from non-D procedures, indicating that ethanol drinking was not related to the operation of the door. Groups that received training procedures providing for both SIO and IS showed more rapid initiation of ethanol intake and more rapid escalation of ethanol intake as the concentration of ethanol in the sipper tube conditioned stimulus was increased across sessions. Theoretical accounts, which are based on cue at response manipulandum/autoshaping, schedule-induced polydipsia, incentive sensitization, and intermittency-induced arousal, are considered.

Effects of drugs that potentiate GABA on extinction of positively-reinforced operant behaviour

Extinction following positively reinforced operant conditioning reduces response frequency, at least in part through the aversive or frustrative effects of non-reinforcement. According to J.A. Gray's theory, non-reinforcement activates the behavioural inhibition system which in turn causes anxiety. As predicted, anxiolytic drugs including benzodiazepines affect the operant extinction process. Recent studies have shown that reducing GABA-mediated neurotransmission retards extinction of aversive conditioning. We have shown in a series of studies that anxiolytic compounds that potentiate GABA facilitate extinction of positively reinforced fixed-ratio operant behaviour in C57B1/6 male mice. This effect does not occur in the early stages of extinction, nor is it dependent on cumulative effects of the compound administered. Potentiation of GABA at later stages has the effect of increasing sensitivity to the extinction contingency and facilitates the inhibition of the behaviour that is no longer required. The GABAergic hypnotic, zolpidem, has the same selective effects on operant extinction in this procedure. The effects of zolpidem are not due to sedative action. There is evidence across our series of experiments that different GABA-A subtype receptors are involved in extinction facilitation and anxiolysis. Consequently, this procedure may not be an appropriate model for anxiolytic drug action, but it may be a useful technique for analysing the neural bases of extinction and designing therapeutic interventions in humans where failure to extinguish inappropriate behaviours can lead to pathological conditions such as post-traumatic stress disorder.

Mathematical principles of reinforcement and resistance to change

Although Killeen's mathematical principles of reinforcement (MPR) apply to the asymptotic rate of a free operant after extended exposure to a single schedule of reinforcement, they can be extended to resistance to change in multiple schedules via alterations in the parameter representing the activating effects of reinforcers. MPR's predictions of resistance to change in relation to reinforcer rate on variable-interval (VI) schedules are empirically correct and agree with behavioral momentum theory (BMT). However, both MPR and BMT encounter problems in accounting for the effects of delayed reinforcement on resistance to change, relative to immediate reinforcement at the same rate. Further problems are raised by differences in resistance to change between variable-ratio (VR) and variable-interval performances maintained by the same reinforcer rate. With both delayed versus immediate reinforcement and variable-ratio versus variable-interval reinforcement, differential resistance to change is negatively correlated with the log ratios of baseline response rates when reinforcer rates are equated. Cases where resistance to change varies despite equated reinforcer rates, and the correlations among behavioral measures, provide challenges and opportunities for both MPR and BMT.

Extending mathematical principles of reinforcement into the domain of behavioral pharmacology

Mathematical principles of reinforcement (MPR) attempts to integrate the empirical laws of reinforcement schedules that have accumulated over the decades. MPR is based on three principles: incentives excite behavior; there are temporal constraints on responding; and coupling of responses to reinforcers strengthens behavior [Behav. Brain Sci. 17 (1994) 105]. In the present paper MPR is extended into the domain of behavioral pharmacology, specifically to model the effects of D-amphetamine on operant behavior. In Experiment 1a, a five-component multiple fixed-ratio schedule was designed to generate behavioral baselines that were subsequently used to assess drug effect. In Experiments 1b and 1c, the quality and quantity of reinforcer were manipulated. The data generated by the three experiments were consistent with MPR. In Experiment 2, MPR was used to model the effects of D-amphetamine on rats responding under the five-component multiple fixed-ratio schedule. According to the model, the rate-decreasing effects of D-amphetamine were due primarily to motor disruption and secondarily to increased impulsivity; at the highest dosages, D-amphetamine also may have decreased the incentive value of food.