MPR

Hyperbolae

Hyperbolic relations between independent and dependent variables are ubiquitous in the experimental analysis of behavior, mentioned in over 150 articles in the Journal of the Experimental Analysis of Behavior. There are two principal forms of hyperbolae: The first describes the relation between response rate and reinforcement rate on variable-interval schedules of reinforcement; it rises asymptotically toward a maximum. The second describes the relation between the current equivalent value of an incentive and its delay or (im)probability; it falls from a maximum toward an asymptote of 0. Where do these come from? What do their parameters mean? How are they related? This article answers the first two questions and addresses the last.

Assessment of the inbred C57BL/6 and outbred CD1 mouse strains using a progressive ratio schedule during development

Inbred strains have a genetic similarity of at least 98.6% compared to their outbred counterparts. Several studies have shown that inbred C57BL/6 mice and outbred ICR (CD1) mice differ in locomotion, cognitive flexibility, and aggression. However, their performance in operant paradigms is not well understood. A progressive ratio (PR) schedule of reinforcement is a method of quantitative estimation of the incentive state of an animal for a reward by increasing response requirements for reinforcer delivery, which is relevant to assess the breakpoint (amount of response effort an animal is willing to invest for a single unit of reward). This study tested male and female C57BL/6 and CD1 mice with an open field to analyze locomotion. Then, we used conditioning chambers with a PR3 schedule for ten consecutive days (P30-P40). PR performance was measured with the breakpoint, and the mathematical principles of reinforcement (MPR) were used to estimate motivation, impulsivity, and motor skills to manipulate the operandum. We found that CD1 mice showed higher locomotor activity than C57BL/6 independently of sex. CD1 mice had a higher breakpoint. However, male CD1 mice gradually increased breakpoint until the last session. In the MPR model, CD1 mice showed decreased fixed paused parameter (impulsivity) than C57BL/6, independent of sex. Our data suggest that the higher breakpoint in CD1 strain may partially be related to impulsivity. Therefore, the MPR model can help identify factors that affect performances, such as motivation, impulsivity, and motor skills during a PR in adolescent CD1 and C57BL/6 mice. These findings are essential to characterize the differences in the behavioral performance between C57BL/6 and CD1 strains and their potential as animal models.

Using rodent data to elucidate dopaminergic mechanisms of ADHD: Implications for human personality

An altered behavioral response to positive reinforcement has been proposed to be a core deficit in attention deficit hyperactivity disorder (ADHD). The spontaneously hypertensive rat (SHR), a congenic animal strain, displays a similarly altered response to reinforcement. The presence of this genetically determined phenotype in a rodent model allows experimental investigation of underlying neural mechanisms. Behaviorally, the SHR displays increased preference for immediate reinforcement, increased sensitivity to individual instances of reinforcement relative to integrated reinforcement history, and a steeper delay of reinforcement gradient compared to other rat strains. The SHR also shows less development of incentive to approach sensory stimuli, or cues, that predict reward after repeated cue-reward pairing. We consider the underlying neural mechanisms for these characteristics. It is well known that midbrain dopamine neurons are initially activated by unexpected reward and gradually transfer their responses to reward-predicting cues. This finding has inspired the dopamine transfer deficit (DTD) hypothesis, which predicts certain behavioral effects that would arise from a deficient transfer of dopamine responses from actual rewards to reward-predicting cues. We argue that the DTD predicts the altered responses to reinforcement seen in the SHR and individuals with ADHD. These altered responses to reinforcement in turn predict core symptoms of ADHD. We also suggest that variations in the degree of dopamine transfer may underlie variations in personality dimensions related to altered reinforcement sensitivity. In doing so, we highlight the value of rodent models to the study of human personality.

Resistance to change, of behavior and of theory

The persistence of operant behavior when disrupted tends to be positively related to how often reinforcers were delivered in the past. Behavioral momentum theory describes this finding as the outcome of Pavlovian processes. That is, the relation between discriminative stimuli and reinforcers that were delivered in their presence strengthens behavior, thereby making it more likely to persist. If only the story were that simple. A growing number of findings challenge the basic tenets of behavioral momentum theory. Some even call into question whether Pavlovian relations contribute to persistence in the first place. In this paper, I will review behavioral momentum theory and some of the data that have been problematic for the theory. I will argue that despite these very real challenges, the theory provides important utility not only to basic analyses of response persistence but also to clinical interventions directed at long-term reductions in problem behavior. It, for example, has set the stage for the development of alternative conceptual analyses of resistance to change, two of which will be highlighted for readers. Moreover, behavioral momentum theory may tell us something important about the reasons it continues to have an influence on the field, despite the challenging data that deter it.

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

Modeling VI and VDRL feedback functions: Searching normative rules through computational simulation

We present the mathematical description of feedback functions of variable interval and variable differential reinforcement of low rates as functions of schedule size only. These results were obtained using an R script named Beak, which was built to simulate rates of behavior interacting with simple schedules of reinforcement. Using Beak, we have simulated data that allow an assessment of different reinforcement feedback functions. This was made with unparalleled precision, as simulations provide huge samples of data and, more importantly, simulated behavior is not changed by the reinforcement it produces. Therefore, we can vary response rates systematically. We've compared different reinforcement feedback functions for random interval schedules, using the following criteria: meaning, precision, parsimony, and generality. Our results indicate that the best feedback function for the random interval schedule was published by Baum (1981). We also propose that the model used by Killeen (1975) is a viable feedback function for the random differential reinforcement of low rates schedule. We argue that Beak paves the way for greater understanding of schedules of reinforcement, addressing still open questions about quantitative features of simple schedules. Also, Beak could guide future experiments that use schedules as theoretical and methodological tools.

Adolescence as a sensitive period for neurotoxicity: Lifespan developmental effects of methylmercury

Neurotoxicity resulting from the environmental contaminant, methylmercury (MeHg), is a source of concern for many human populations that rely heavily on the consumption of fish and rice as stable ingredients in the diet. The developmental period of exposure is important both to the qualitative effects of MeHg and to the dose required to produce those effects. MeHg exposure during the sensitive prenatal period causes deleterious and long-lasting changes in neurodevelopment at particularly low doses. The effects include a wide host of cognitive and behavioral outcomes expressed in adulthood and sometimes not until aging. However, neurotoxic outcomes of methylmercury when exposure occurs during adolescence are only recently revealing impacts on human populations and animal models. This review examines the current body of work and showcases the sensitivity of adolescence, a period that straddles early development and adulthood, to methylmercury neurotoxicity and the implications such toxicity has in our understanding of methylmercury's effects in human populations and animal models.

Ensuring Effective Public Health Communication: Insights and Modeling Efforts From Theories of Behavioral Economics, Heuristics, and Behavioral Analysis for Decision Making Under Risk

Public health (PH) messaging can have an enormous impact on shaping how individuals within society behave, and can ensure it is in a safe and responsible way, consistent with up-to-date evidence-based PH guidelines. If done effectively, messaging can save lives and improve the health of those within society. However, unfortunately, those within Government PH bodies typically have little training about how to effectively represent PH messages in a way that is consistent with psychological theories of cognitive bias, in order to avoid cognitively biasing the public through their messages. As a result of this, inadequate representation of PH messages can result, which can often lead to cognitive bias in those from the public who read or listen to the message information. This can lead to poor decision making of the pubic as a whole, which can then further lead to harm and even death of public members as a result of these poor decisions. One way to minimize the problem of bias in decision making is to explore psychology theories that model how bias can occur from PH messaging, and identify ways in which PH agencies can utilize such approaches to improve the effectiveness of their messages. Previous focus has been largely on behavioral economic theories, however, here, other accounts are offered in addition to these. These include theories of heuristics and theories from the behavior analysis domain, which may increase the predictive power of modeling bias, and have applications for how best to represent PH message information which minimize bias.

Longer operant lever-press duration requirements induce fewer but longer response bouts in rats

Operant behavior is organized in bouts that are particularly visible under variable-interval (VI) schedules of reinforcement. Previous research showed that increasing the work required to produce a response decreases the rate at which bouts are emitted and increases the minimum interresponse time (IRT). In the current study, the minimum effective IRT was directly manipulated by changing the minimum duration of effective lever presses reinforced on a VI 40-s schedule. Contrary to assumptions of previous models, response durations were variable. Response durations were typically 0.5 s greater than the minimum duration threshold; durations that exceeded this threshold were approximately log-normally distributed. As the required duration threshold increased, rats emitted fewer but longer bouts. This effect may reflect an effort-induced reduction in motivation and a duration-induced facilitation of a response-outcome association.

Matching Behaviours and Rewards

Matching describes how behaviour is related to rewards. The matching law holds when the ratio of an individual's behaviours equals the ratio of the rewards obtained. From its origins in the study of pigeons working for food in the laboratory, the law has been applied to a range of species, both in the laboratory and outside it (e.g., human sporting decisions). Probability matching occurs when the probability of a behaviour equals the probability of being rewarded. Input matching predicts the distribution of individuals across habitats. We evaluate the rationality of the matching law and probability matching, expose the logic of matching in real-world cases, review how recent neuroscience findings relate to matching, and suggest future research directions.

Tactical contingencies in the experimental analysis of reinforcement and operant classes

In his "Tactics of Scientific Research" (1960), his work on avoidance, his discovery of equivalence classes and his cautions on applications of coercion, Murray Sidman created high standards for behavior analytic research. I illustrate his influence in the context of three examples he might have characterized as pilot studies. Each examined trial N+1 response probabilities depending on whether trial N responding had produced a reinforcer. Differentially reinforced interresponse times, keys pecked in arbitrary matching, and two-key response sequences provided no robust evidence that reinforcing some response property on trial N raises the probability of responding with that property on trial N+1. These negative findings shed light on the nature of operant classes and on the relation of reinforcers to the responses that produced them. Through selection, reinforcers create operant classes and engender variations of the responses within those classes; operant classes are held together by common contingencies. Sidman extended our understanding of operant classes by expanding them to include equivalence relations.

A computational formulation of the behavior systems account of the temporal organization of motivated behavior

The behavior systems framework suggests that motivated behavior-e.g., seeking food and mates, avoiding predators-consists of sequences of actions organized within nested behavioral states. This framework has bridged behavioral ecology and experimental psychology, providing key insights into critical behavioral processes. In particular, the behavior systems framework entails a particular organization of behavior over time. The present paper examines whether such organization emerges from a generic Markov process, where the current behavioral state determines the probability distribution of subsequent behavioral states. This proposition is developed as a systematic examination of increasingly complex Markov models, seeking a computational formulation that balances adherence to the behavior systems approach, parsimony, and conformity to data. As a result of this exercise, a nonstationary partially hidden Markov model is selected as a computational formulation of the predatory subsystem. It is noted that the temporal distribution of discrete responses may further unveil the structure and parameters of the model but, without proper mathematical modeling, these discrete responses may be misleading. Opportunities for further elaboration of the proposed computational formulation are identified, including developments in its architecture, extensions to defensive and reproductive subsystems, and methodological refinements.

Cocaine-associated decision-making: Toward isolating preference

Ever-increasing evidence suggests that substance use disorder is mediated by decision-making processes, and as such, providing nondrug alternatives can shift maladaptive preferences away from drug reinforcers, such as cocaine. Of note, a recent hypothesis suggests that preference for cocaine is simply a byproduct of cocaine intake, such that the 'direct' effects of cocaine weaken the impact of non-drug alternatives while measuring choice. Conversely, existing quantitative theories of decision-making suggest preference is determined by various dimensions of concurrent reinforcers that in turn determine the relative value of available alternatives. Toward teasing apart the conflicting theories above, we developed a novel drug-choice procedure to control for reinforcer frequency and magnitude (two reinforcer dimensions well known to influence preference) that consequently controls for overall cocaine intake. As predicted by quantitative choice theory, results suggest that cocaine intake and preference are dissociable while measuring choice, with reinforcer frequency and magnitude having independent influence on the relative value of choice alternatives. Furthermore, we demonstrate that the choice procedure is sensitive to various manipulations known to alter cocaine reinforcement, all while keeping cocaine intake constant. Finally, the results point to the process of economic substitution as an important avenue of future neurobehavioral investigation toward the improvement of behavioral and pharmacological therapies for substance use disorders. Overall, the proposed choice procedure will allow for improved isolation of the neurobehavioral processes that mediate drug-associated decision-making in future studies.

Reinforcer quality matters: A test of the Mathematical Principles of Reinforcement with domestic hens (Gallus gallus domesticus)

This study evaluated the ability of Killeen's (1994) Mathematical Principles of Reinforcement to account for the effects of changes in reinforcer quality on hens' rates of responding on fixed-ratio schedules. Hens were trained to peck a key on a fixed-ratio schedule of reinforcement and then experienced an ascending series of ratio values in two separate conditions. In different conditions, the food reinforcer was either wheat or puffed wheat. Response rates initially increased with increases in ratio requirement before eventually decreasing at larger ratios. Quantitative fits of the model accounted for the data well. The fits revealed that different foods were systematically associated with changes in the specific activation parameter, a, and these were consistent with previous reports of preference for those food items.

d-Amphetamine and methylmercury exposure during adolescence alters sensitivity to monoamine uptake inhibitors in adult mice

Gestational exposure to methylmercury (MeHg), an environmental neurotoxicant, and adolescent administration of d-amphetamine (d-AMP) disrupt dopamine neurotransmission and alter voluntary behavior in adult rodents. We determined the impact of adolescent exposure to MeHg and d-AMP on monoamine neurotransmission in mice by assessing sensitivity to acute d-AMP, desipramine, and clomipramine, drugs that target dopamine, norepinephrine, and serotonin reuptake, respectively. Male C57Bl/6n mice were given 0 (control) or 3 ppm MeHg via drinking water from postnatal day 21 to 60 (murine adolescence). Within each group, mice were given once-daily injections of d-AMP or saline (i.p.) from postnatal day 28 to 42. This exposure regimen produced four treatment groups (n = 10-12/group): control, d-AMP, MeHg, and d-AMP + MeHg. As adults, the mice lever pressed under fixed-ratio schedules of reinforcement (FR 1, 5, 15, 30, 60, and 120). Acute i.p. injections of d-AMP (.3-1.7 mg/kg), desipramine (5.6-30 mg/kg), and clomipramine (5.6-30 mg/kg) were administered in adulthood after a stable behavioral baseline was established. Adolescent MeHg exposure increased saturation rate and minimum response time, an effect that was mitigated by chronic administration of d-AMP in adolescence. In unexposed mice, the three monoamine reuptake inhibitors had separable behavioral effects. Adolescent d-AMP increased sensitivity to acute d-AMP, desipramine, and clomipramine. Adolescent MeHg exposure alone did not alter drug sensitivity. Combined adolescent d-AMP + MeHg exposure enhanced sensitivity to acute d-AMP's and desipramine's effects on minimum response time. Adolescence is a vulnerable developmental period during which exposure to chemicals can have lasting effects on monoamine function and behavior.

Dorsal striatal dopamine D1 receptor availability predicts an instrumental bias in action learning

The brain’s dopaminergic pathways are crucially important for adaptive behavior. They are thought to enable us to approach rewards and stay away from punishments. During learning, dopaminergic reward prediction errors are thought to reinforce previously rewarded actions, so they become easier to repeat. This dopaminergic activity could lead to a systematic bias by which rewarded actions are more readily learned than rewarded inactions. We present two findings. First, dopamine receptors in cortex, dorsal striatum, and nucleus accumbens provide distinct sources of variance in the human brain. Second, the boost in an individual’s learning rate from previously rewarded actions is dependent on the dopamine receptor density in dorsal striatum, a central structure in the dopaminergic circuit.

Learning to act to obtain reward and inhibit to avoid punishment is easier compared with learning the opposite contingencies. This coupling of action and valence is often thought of as a Pavlovian bias, although recent research has shown it may also emerge through instrumental mechanisms. We measured this learning bias with a rewarded go/no-go task in 60 adults of different ages. Using computational modeling, we characterized the bias as being instrumental. To assess the role of endogenous dopamine (DA) in the expression of this bias, we quantified DA D1 receptor availability using positron emission tomography (PET) with the radioligand [¹¹C]SCH23390. Using principal-component analysis on the binding potentials in a number of cortical and striatal regions of interest, we demonstrated that cortical, dorsal striatal, and ventral striatal areas provide independent sources of variance in DA D1 receptor availability. Interindividual variation in the dorsal striatal component was related to the strength of the instrumental bias during learning. These data suggest at least three anatomical sources of variance in DA D1 receptor availability separable using PET in humans, and we provide evidence that human dorsal striatal DA D1 receptors are involved in the modulation of instrumental learning biases.

Adolescent methylmercury exposure: Behavioral mechanisms and effects of sodium butyrate in mice

Methylmercury (MeHg), an environmental neurotoxicant primarily found in fish, produces neurobehavioral impairment when exposure occurs during gestation. Whether other developmental periods, such as adolescence, display enhanced vulnerability to the behavioral effects of MeHg exposure is only beginning to be explored. Further, little is known about the effects of repeated administration of lysine deacetylase inhibitors, such as sodium butyrate (NaB), on operant behavior. In Experiment 1, male C57BL6/n mice were exposed to 0, 0.3, and 3.0 ppm MeHg (n = 12 each) via drinking water from postnatal days 21 to 60 (murine adolescence). As adults, mice were trained to lever press under an ascending series of fixed-ratio schedules of milk reinforcement selected to enable the analysis of three important parameters of operant behavior using the framework provided by Mathematical Principles of Reinforcement. Adolescent MeHg exposure dose-dependently increased saturation rate, a measure of the retroactive reach of a reinforcer, and decreased minimum response time relative to controls. In Experiment 2, the behavioral effects of repeated NaB administration both alone and following adolescent MeHg exposure were examined. Male C57BL6/n mice were given either 0 or 3.0 ppm MeHg during adolescence and, before behavioral testing, two weeks of once daily i.p. injections of saline or 0.6 g/kg NaB (n = 12 in each cell). Adolescent MeHg exposure again increased saturation rate but did not significantly alter minimum response time. NaB also increased saturation rate in both MeHg exposure groups. These data suggest that the behavioral mechanisms of adolescent MeHg exposure and NaB may be related to the impact of reinforcement on prior responses. Specifically, MeHg and NaB concentrated the effects of reinforcers onto the most recent responses.

Race, Offense Type, and Suicide Ideation: Tests of the Interpersonal-Psychological Theory in Juvenile Offenders

This study evaluated the synergy hypothesis of the Interpersonal-Psychological Theory of Suicide (IPTS), which argues thwarted belongingness and perceived burdensomeness are positively interactive in their association with suicide ideation, in a group of juvenile offenders. It also examined whether this prediction is differentially applicable across race/ethnicity or offense type. Participants included 590 adjudicated and confined male juveniles. Regression was used to test the association between suicide ideation and thwarted belongingness, perceived burdensomeness, and their interaction term. Subsequent analyses included tests of group interactions related to race/ethnicity and offense type. No interaction between thwarted belongingness and perceived burdensomeness was observed, despite adequate power. No significant group interactions were observed for race/ethnicity or offense type. However, results did show significant linear relationships between thwarted belongingness, perceived burdensomeness, and ideation, highlighting their potential utility as intervention targets in this at-risk population. Thus, although the current results are the first to show the basic IPTS risk factors generalize across race/ethnicity and offense type, they also failed to support that those factors were interactive, a primary IPTS claim. The absence of an interaction between thwarted belongingness and perceived burdensomeness suggests their role in suicide ideation for juvenile offenders may be more parsimonious than the IPTS proposes.

Optimal response vigor and choice under non-stationary outcome values

Within a rational framework, a decision-maker selects actions based on the reward-maximization principle, which stipulates that they acquire outcomes with the highest value at the lowest cost. Action selection can be divided into two dimensions: selecting an action from various alternatives, and choosing its vigor, i.e., how fast the selected action should be executed. Both of these dimensions depend on the values of outcomes, which are often affected as more outcomes are consumed together with their associated actions. Despite this, previous research has only addressed the computational substrate of optimal actions in the specific condition that the values of outcomes are constant. It is not known what actions are optimal when the values of outcomes are non-stationary. Here, based on an optimal control framework, we derive a computational model for optimal actions when outcome values are non-stationary. The results imply that, even when the values of outcomes are changing, the optimal response rate is constant rather than decreasing. This finding shows that, in contrast to previous theories, commonly observed changes in action rate cannot be attributed solely to changes in outcome value. We then prove that this observation can be explained based on uncertainty about temporal horizons; e.g., the session duration. We further show that, when multiple outcomes are available, the model explains probability matching as well as maximization strategies. The model therefore provides a quantitative analysis of optimal action and explicit predictions for future testing.

Suppressive and enhancing effects of nicotine on food-seeking behavior

The present study examined how systemic low doses of nicotine affect the microstructure of reinforced food-seeking behavior in rats. Rats were first given an acute saline or nicotine treatment (0.1-0.6mg/kg, with an inter-injection interval of at least 48h), and then a chronic saline or nicotine treatment (0.3mg/kg/day for 10 consecutive days). Immediately after each injection, rats were required to press a lever five times to obtain food that was available at unpredictable times (on average every 80s) with constant probability. Acute nicotine dose-dependently suppressed behavior prior to the delivery of the first reinforcer, but enhanced food-reinforced behavior afterwards. These effects were primarily observed in the time it took rats to initiate food-seeking behavior. Enhancing effects were also observed in the microstructure of food-seeking behavior, with lower nicotine doses (0.1-0.3mg/kg) increasing the rate at which response bouts were initiated, and higher doses (0.3-0.6mg/kg) increasing within-bout response rates. A pre-feeding control suggests that changes in appetite alone cannot explain these effects. Over the course of chronic nicotine exposure, tolerance developed to the suppressive, but not to the enhancing effects of nicotine on food-seeking behavior. These results suggest that (a) lower doses of nicotine enhance the reward value of food and/or food-associated stimuli, (b) higher doses of nicotine enhance motoric activity, and (c) ostensive sensitization effects of nicotine on behavior partially reflect a tolerance to its transient suppressive motoric effects.

Human instrumental performance in ratio and interval contingencies: A challenge for associative theory

Associative learning theories regard the probability of reinforcement as the critical factor determining responding. However, the role of this factor in instrumental conditioning is not completely clear. In fact, free-operant experiments show that participants respond at a higher rate on variable ratio than on variable interval schedules even though the reinforcement probability is matched between the schedules. This difference has been attributed to the differential reinforcement of long inter-response times (IRTs) by interval schedules, which acts to slow responding. In the present study, we used a novel experimental design to investigate human responding under random ratio (RR) and regulated probability interval (RPI) schedules, a type of interval schedule that sets a reinforcement probability independently of the IRT duration. Participants responded on each type of schedule before a final choice test in which they distributed responding between two schedules similar to those experienced during training. Although response rates did not differ during training, the participants responded at a lower rate on the RPI schedule than on the matched RR schedule during the choice test. This preference cannot be attributed to a higher probability of reinforcement for long IRTs and questions the idea that similar associative processes underlie classical and instrumental conditioning.

On the time course of rapid and repeatable response induction

As reinforcers are delivered for a particular response, that response increases. Other similar responses also increase, with the extent to which this increase is seen being positively related to the similarity between the responses. This process, called induction, is pronounced during initial response acquisition, and dissipates as the target response is established. The present experiment examined this process in a paradigm wherein responses to one of five nose poke receptacles results in reinforcement for six sessions in each condition and the target location changes between conditions. Consistent patterns of induction were seen across conditions, with patterns of induction dissipating within each condition.

Spatial discrimination reversal and incremental repeated acquisition in adolescent and adult BALB/c mice

Adolescence is characterized by neural and behavior development that includes increases in novel experiences and impulsive choice. Experimental rodent models can characterize behavior phenotypes that typify adolescence. The present experiment was designed to characterize differences between adolescent (post-natal day (PND) 34-60) and adult (PND 70-96) BALB/c mice using a response-initiated spatial discrimination reversal (SDR) and incremental repeated acquisition of response chains (IRA) procedures. During SDR, adolescents omitted more trials and were slower to initiate trials than adults, but the age groups did not differ on accuracy and perseveration measures. During IRA, adolescents displayed poorer overall performance (measured by progress quotient), lower accuracy at individual chain links, and completed fewer long response chains (>3 links) than adults. In both procedures (SDR and IRA), the poorer performance of adolescents appeared to be related to the use of a response device that was spatially removed from reinforcer delivery. These results indicate that SDR and IRA performance can be established during the brief rodent adolescent period but that these two age groups' performances differ. We hypothesize that adolescent behavior is more sensitive than adult behavior to the spatiotemporal distance between response device and location of reinforcer delivery.

A quantitative analysis of the effects of qualitatively different reinforcers on fixed ratio responding in inbred strains of mice

Previous studies of inbred mouse strains have shown reinforcer-strain interactions that may potentially mask differences among strains in memory performance. The present research examined the effects of two qualitatively different reinforcers (heterogeneous mix of flavored pellets and sweetened-condensed milk) on responding maintained by fixed-ratio schedules of reinforcement in three inbred strains of mice (BALB/c, C57BL/6, and DBA/2). Responses rates for all strains were a bitonic (inverted U) function of the size of the fixed-ratio schedule and were generally higher when responding was maintained by milk. For the DBA/2 and C57BL/6 and to a lesser extent the BALB/c, milk primarily increased response rates at moderate fixed ratios, but not at the largest fixed ratios tested. A formal model of ratio-schedule performance, Mathematical Principles of Reinforcement (MPR), was applied to the response rate functions of individual mice. According to MPR, the differences in response rates maintained by pellets and milk were mostly due to changes in motoric processes as indicated by changes in the minimum response time (δ) produced by each reinforcer type and not specific activation (a), a model term that represents value and is correlated with reinforcer magnitude and the break point obtained under progressive ratio schedules. MPR also revealed that, although affected by reinforcer type, a parameter interpreted as the rate of saturation of working memory (λ), differed among the strains.

Effects of SKF-83566 and haloperidol on performance on progressive ratio schedules maintained by sucrose and corn oil reinforcement: quantitative analysis using a new model derived from the Mathematical Principles of Reinforcement (MPR)

RATIONALE

Mathematical models can assist the interpretation of the effects of interventions on schedule-controlled behaviour and help to differentiate between processes that may be confounded in traditional performance measures such as response rate and the breakpoint in progressive ratio (PR) schedules.

OBJECTIVE

The effects of a D1-like dopamine receptor antagonist, 8-bromo-2,3,4,5-tetrahydro-3-methyl-5-phenyl-1H-3-benzazepin-7-ol hydrobromide (SKF-83566), and a D2-like receptor antagonist, haloperidol, on rats' performance on PR schedules maintained by sucrose and corn oil reinforcers were assessed using a new model derived from Killeen's (Behav Brain Sci 17:105-172, 1994) Mathematical Principles of Reinforcement.

METHOD

Separate groups of rats were trained under a PR schedule using sucrose or corn oil reinforcers. SKF-83566 (0.015 and 0.03 mg kg(-1)) and haloperidol (0.05 and 0.1 mg kg(-1)) were administered intraperitoneally (five administrations of each treatment). Running and overall response rates in successive ratios were analysed using the new model, and estimates of the model's parameters were compared between treatments.

RESULTS

Haloperidol reduced a (the parameter expressing incentive value) in the case of both reinforcers, but did not affect the parameters related to response time and post-reinforcement pausing. SKF-83566 reduced a and k (the parameter expressing sensitivity of post-reinforcement pausing to the prior inter-reinforcement interval) in the case of sucrose, but did not affect any of the parameters in the case of corn oil.

CONCLUSIONS

The results are consistent with the hypothesis that blockade of both D1-like and D2-like receptors reduces the incentive value of sucrose, whereas the incentive value of corn oil is more sensitive to blockade of D2-like than D1-like receptors.

Discrimination of variable schedules is controlled by interresponse times proximal to reinforcement

In Experiment 1, food-deprived rats responded to one of two schedules that were, with equal probability, associated with a sample lever. One schedule was always variable ratio, while the other schedule, depending on the trial within a session, was: (a) a variable-interval schedule; (b) a tandem variable-interval, differential-reinforcement-of-low-rate schedule; or (c) a tandem variable-interval, differential-reinforcement-of-high-rate schedule. Completion of a sample-lever schedule, which took approximately the same time regardless of schedule, presented two comparison levers, one associated with each sample-lever schedule. Pressing the comparison lever associated with the schedule just presented produced food, while pressing the other produced a blackout. Conditional-discrimination accuracy was related to the size of the difference in reinforced interresponse times and those that preceded it (predecessor interresponse times) between the variable-ratio and other comparison schedules. In Experiment 2, control by predecessor interresponse times was accentuated by requiring rats to discriminate between a variable-ratio schedule and a tandem schedule that required emission of a sequence of a long, then a short interresponse time in the tandem's terminal schedule. These discrimination data are compatible with the copyist model from Tanno and Silberberg (2012) in which response rates are determined by the succession of interresponse times between reinforcers weighted so that each interresponse time's role in rate determination diminishes exponentially as a function of its distance from reinforcement.

A theory of behaviour on progressive ratio schedules, with applications in behavioural pharmacology

RATIONALE

Mathematical principles of reinforcement (MPR) provide the theoretical basis for a family of models of schedule-controlled behaviour. A model of fixed-ratio schedule performance that was applied to behaviour on progressive ratio (PR) schedules showed systematic departures from the data.

OBJECTIVE

This study aims to derive a new model from MPR that will account for overall and running response rates in the component ratios of PR schedules, and their decline toward 0, the breakpoint.

METHOD

The role of pausing is represented in a real-time model containing four parameters: T (0) and k are the intercept and slope of the linear relation between post-reinforcement pause duration and the prior inter-reinforcer interval; a (specific activation) measures the incentive value of the reinforcer; δ (response time) sets biomechanical limits on response rate. Running rate is predicted to decrease with negative acceleration as ratio size increments, overall rate to increase and then decrease. Differences due to type of progression are explained as hysteresis in the control by reinforcement rates. Re-analysis of extant data focuses on the effects of acute treatment with antipsychotic drugs, lesions of the nucleus accumbens core, and destruction of orexinergic neurones of the lateral hypothalamus.

RESULTS

The new model resolves some anomalies evident in earlier analyses, and provides new insights to the results of these interventions.

CONCLUSIONS

Because they can render biologically relevant parameters, mathematical models can provide greater power in interpreting the effects of interventions on the processes underlying schedule-controlled behaviour than is possible for first-order data such as the breakpoint.

A clozapine-like effect of cyproheptadine on progressive ratio schedule performance

The atypical antipsychotic drug clozapine has multiple pharmacological actions, some of which, including 5-hydroxytryptamine (5-HT₂) and histamine (H₁) receptor antagonist effects, are shared by the non-selective 5-HT receptor antagonist cyproheptadine. Atypical antipsychotics have a characteristic profile of action on operant behaviour maintained by progressive ratio schedules, as revealed by Killeen's (1994) mathematical model of schedule controlled behaviour. These drugs increase the values of a parameter that expresses the 'incentive value' of the reinforcer (a) and a parameter that is inversely related to the 'motor capacity' of the organism (δ). This experiment examined the effects of acute treatment with cyproheptadine and clozapine on performance on a progressive ratio schedule of food reinforcement in rats; the effects of a conventional antipsychotic, haloperidol, and two drugs with food intake-enhancing effects, chlordiazepoxide and Δ⁹-tetrahydrocannabinol (THC), were also examined. Cyproheptadine (1, 5 mg kg⁻¹) and clozapine (3.75, 7.5 mg kg⁻¹) increased a and δ. Haloperidol (0.05, 0.1 mg kg⁻¹) reduced a and increased δ. Chlordiazepoxide (3, 10 mg kg⁻¹) increased a but reduced δ. THC (1, 3 mg kg⁻¹) had no effect. Interpretation based on Killeen's (1994) model suggests that cyproheptadine and clozapine enhanced the incentive value of the reinforcer and impaired motor performance. Motor impairment may be due to sedation (possibly reflecting H₁ receptor blockade). Enhancement of incentive value may reflect simultaneous blockade of H₁ and 5-HT₂ receptors, which has been proposed as the mechanism underlying the food intake-enhancing effect of cyproheptadine. In agreement with previous findings, haloperidol impaired motor performance and reduced the incentive value of the reinforcer. Chlordiazepoxide's effect on a is consistent with its food intake-enhancing effect.

Effect of orexin-B-saporin-induced lesions of the lateral hypothalamus on performance on a progressive ratio schedule

It has been suggested that a sub-population of orexinergic neurones whose somata lie in the lateral hypothalamic area (LHA) play an important role in regulating the reinforcing value of both food and drugs. This experiment examined the effect of disruption of orexinergic mechanisms in the LHA on performance on the progressive ratio schedule of reinforcement, in which the response requirement increases progressively for successive reinforcers. The data were analysed using a mathematical model which yields a quantitative index of reinforcer value and dissociates effects of interventions on motor and motivational processes. Rats were trained under a progressive ratio schedule using food-pellet reinforcement. They received bilateral injections of conjugated orexin-B-saporin (OxSap) into the LHA or sham lesions. Training continued for a further 40 sessions after surgery. Equations were fitted to the response rate data from each rat, and the parameters of the model were derived for successive blocks of 10 sessions. The OxSap lesion reduced the number of orexin-containing neurones in the LHA by approximately 50% compared with the sham-lesioned group. The parameter expressing the incentive value of the reinforcer was not significantly altered by the lesion. However, the parameter related to the maximum response rate was significantly affected, suggesting that motor capacity was diminished in the OxSap-lesioned group. The results indicate that OxSap lesions of the LHA disrupted food-reinforced responding on the progressive ratio schedule. It is suggested that this disruption was brought about by a change in non-motivational (motor) processes.

Extinction under a behavioral microscope: isolating the sources of decline in operant response rate

Extinction performance is often used to assess underlying psychological processes without the interference of reinforcement. For example, in the extinction/reinstatement paradigm, motivation to seek drug is assessed by measuring responding elicited by drug-associated cues without drug reinforcement. However, extinction performance is governed by several psychological processes that involve motivation, memory, learning, and motoric functions. These processes are confounded when overall response rate is used to measure performance. Based on evidence that operant responding occurs in bouts, this paper proposes an analytic procedure that separates extinction performance into several behavioral components: (1-3) the baseline bout initiation rate, within-bout response rate, and bout length at the onset of extinction; (4-6) their rates of decay during extinction; (7) the time between extinction onset and the decline of responding; (8) the asymptotic response rate at the end of extinction; (9) the refractory period after each response. Data that illustrate the goodness of fit of this analytic model are presented. This paper also describes procedures to isolate behavioral components contributing to extinction performance and make inferences about experimental effects on these components. This microscopic behavioral analysis allows the mapping of different psychological processes to distinct behavioral components implicated in extinction performance, which may further our understanding of the psychological effects of neurobiological treatments.

The isolation of motivational, motoric, and schedule effects on operant performance: a modeling approach

Dissociating motoric and motivational effects of pharmacological manipulations on operant behavior is a substantial challenge. To address this problem, we applied a response-bout analysis to data from rats trained to lever press for sucrose on variable-interval (VI) schedules of reinforcement. Motoric, motivational, and schedule factors (effort requirement, deprivation level, and schedule requirements, respectively) were manipulated. Bout analysis found that interresponse times (IRTs) were described by a mixture of two exponential distributions, one characterizing IRTs within response bouts, another characterizing intervals between bouts. Increasing effort requirement lengthened the shortest IRT (the refractory period between responses). Adding a ratio requirement increased the length and density of response bouts. Both manipulations also decreased the bout-initiation rate. In contrast, food deprivation only increased the bout-initiation rate. Changes in the distribution of IRTs over time showed that responses during extinction were also emitted in bouts, and that the decrease in response rate was primarily due to progressively longer intervals between bouts. Taken together, these results suggest that changes in the refractory period indicate motoric effects, whereas selective alterations in bout initiation rate indicate incentive-motivational effects. These findings support the use of response-bout analyses to identify the influence of pharmacological manipulations on processes underlying operant performance.

Early-life seizures produce lasting alterations in the structure and function of the prefrontal cortex

Early-life seizures (ELS) are associated with long-term behavioral disorders including autism and ADHD, suggesting that frontal lobe structures may be permanently affected. We tested whether ELS produce structural alterations in the prefrontal cortex (PFC) and impair PFC-mediated function using an operant task of behavioral flexibility in rats. Adult rats that had been exposed to 75 flurothyl seizures during postnatal days 1-10 showed decreased behavioral flexibility in the task compared to controls over multiple behavioral sessions, measured as a lever preference asymmetry (p<0.001) and a decreased efficiency of attaining food rewards (p<0.05). ELS rats also showed an increased thickness of the PFC (p<0.01), primarily attributed to layer V (p<0.01) with no differences in cell density. These structural changes correlated with lever preference behavioral impairments (p<0.05). This study demonstrates that the consequences of ELS extend to the PFC, which may help explain the high prevalence of comorbid behavioral disorders following ELS.

The ability of lambs to form expectations and the emotional consequences of a discrepancy from their expectations

Appraisal theories developed in cognitive psychology are used here to attempt to better understand emotional experiences in animals. We investigated whether lambs are able to form expectations and whether their emotional responses are affected by situations discrepant from the expectations they may have formed. Forty-five female lambs were trained to obtain a small or a large amount of food reward by performing an operant task (introducing their muzzle into a hole). Then, half the lambs were shifted to the large or the small reward (i.e. positive or negative shift respectively), while the remaining half continued to get the same amount of reward. Thereafter, the lambs previously submitted to a reward change were shifted back to their initial amount of reward (i.e. successive shifts) while the lambs previously maintained on the same amount of reward were subjected to extinction (no reward, thus a negative shift). Behavior, cortisol levels and cardiac activity were analyzed, and the treatments were compared with ANOVAs for mixed models. When the amount of reward delivered was decreased, the lambs showed more locomotor activity and performed the operant task at a higher frequency but less efficiently, and there was a decrease in the parasympathetic influence on their cardiac activity. These responses were exacerbated when the negative shift followed a positive one. Similar responses were observed under extinction, and these responses were more pronounced when animals were trained with a large amount of reward before extinction. In response to a positive shift, we noticed a decrease in the frequency of the attempted operant task; this occurred only when the positive shift followed a negative one. Variations in plasma cortisol were not consistent with changes in the amount of reward. This study shows that lambs evaluate a reward according to their previous experience with that reward. They are able to form expectations, and a discrepancy from these expectations influences emotional responses, especially in the case of a negative shift. Given the appraisal criteria used by lambs and the matching emotions, we can assume that the emotional response to a negative shift expressed by lambs could reflect the despair caused by frustration.

Repeated extinction and reversal learning of an approach response supports an arousal-mediated learning model

We assessed the effects of repeated extinction and reversals of two conditional stimuli (CS+/CS-) on an appetitive conditioned approach response in rats. Three results were observed that could not be accounted for by a simple linear operator model such as the one proposed by Rescorla and Wagner (1972): (1) responding to a CS- declined faster when a CS+ was simultaneously extinguished; (2) reacquisition of pre-extinction performance recovered rapidly within one session; and (3) reversal of CS+/CS- contingencies resulted in a more rapid recovery to the current CS- (former CS+) than the current CS+, accompanied by a slower acquisition of performance to the current CS+. An arousal parameter that mediates learning was introduced to a linear operator model to account for these effects. The arousal-mediated learning model adequately fit the data and predicted data from a second experiment with different rats in which only repeated reversals of CS+/CS- were assessed. According to this arousal-mediated learning model, learning is accelerated by US-elicited arousal and it slows down in the absence of US. Because arousal varies faster than conditioning, the model accounts for the decline in responding during extinction mainly through a reduction in arousal, not a change in learning. By preserving learning during extinction, the model is able to account for relapse effects like rapid reacquisition, renewal, and reinstatement.

Orexin signaling via the orexin 1 receptor mediates operant responding for food reinforcement

BACKGROUND

Orexin (hypocretin) signaling is implicated in drug addiction and reward, but its role in feeding and food-motivated behavior remains unclear.

METHODS

We investigated orexin's contribution to food-reinforced instrumental responding using an orexin 1 receptor (Ox1r) antagonist, orexin -/- (OKO) and littermate wildtype (WT) mice, and RNAi-mediated knockdown of orexin. C57BL/6J (n = 76) and OKO (n = 39) mice were trained to nose poke for food under a variable ratio schedule of reinforcement. After responding stabilized, a progressive ratio schedule was initiated to evaluate motivation to obtain food reinforcement.

RESULTS

Blockade of Ox1r in C57BL/6J mice impaired performance under both the variable ratio and progressive ratio schedules of reinforcement, indicating impaired motivational processes. In contrast, OKO mice initially demonstrated a delay in acquisition but eventually achieved levels of responding similar to those observed in WT animals. Moreover, OKO mice did not differ from WT mice under a progressive ratio schedule, indicating delayed learning processes but no motivational impairments. Considering the differences between pharmacologic blockade of Ox1r and the OKO mice, animals with RNAi mediated knockdown of orexin were then generated and analyzed to eliminate possible developmental effects of missing orexin. Orexin gene knockdown in the lateral hypothalamus in C57BL/6J mice resulted in blunted performance under both the variable ratio and progressive ratio schedules, resembling data obtained following Ox1r antagonism.

CONCLUSIONS

The behavior seen in OKO mice likely reflects developmental compensation often seen in mutant animals. These data suggest that activation of the Ox1r is a necessary component of food-reinforced responding, motivation, or both in normal mice.

Progressive ratio schedules of reinforcement

Pigeons' pecks produced grain under progressive ratio (PR) schedules, whose response requirements increased systematically within sessions. Experiment 1 compared arithmetic (AP) and geometric (GP) progressions. Response rates increased as a function of the component ratio requirement, then decreased linearly (AP) or asymptotically (GP). Experiment 2 found the linear decrease in AP rates to be relatively independent of step size. Experiment 3 showed pausing to be controlled by the prior component length, which predicted the differences between PR and regressive ratio schedules found in Experiment 4. When the longest component ratios were signaled by different key colors, rates at moderate ratios increased, demonstrating control by forthcoming context. Models for response rate and pause duration based on Bizo and Killeen (1997) described performance on AP schedules; GP schedules required an additional parameter representing the contextual reinforcement.