Reinforcer accumulation in a token-reinforcement context with pigeons

Bidding for delayed rewards: Accumulation as delay discounting, delay discounting as regulation, demand functions as corollary

In the accumulation paradigm animals press one manipulandum to accumulate pellets or seconds of access to food, and then press another manipulandum, or run some distance, to collect it. The accumulation may be interpreted as delay discounting, with the animals adjusting the distal amount to compensate for its distance or delay. The amount accumulated before being collected is a linear function of the distance or time that the experimental paradigm stipulates for collecting it. That linear function follows from the Unit-Price/Unit-Amount axiom. The inverse of the linear compensation functions gives a delay-discount hyperbola. The advantages of the accumulation paradigm and analytic framework for delay discounting studies are noted. Compensation functions are then derived from a behavioral regulation model, which generalizes them to contexts where the individual's budget for response cost becomes over-taxed. In turn, such compensation/regulation models lead directly to representative demand functions. In sum, regulation models provide a theoretical grounding for demand functions, compensation functions, and delay discounting hyperbolas. The parameter that links them is the unit amount k, the slope of the compensation function and of the discount function, the setpoint for consumption in a regulation model, and the ideal quantity consumed at minimal price in a demand analysis.

The role of temporal intervals on reinforcer accumulation

Animals accumulate reinforcers when they forgo the opportunity to consume available food in favor of acquiring additional food for later consumption. Laboratory research has shown that reinforcer accumulation is facilitated when an interval (either spatial or temporal) separates earning from consuming reinforcers. However, there has been no systematic investigation on the interval separating consuming reinforcers from earning additional reinforcers. This oversight is problematic because this second interval is an integral part of much of the previous research on reinforcer accumulation. The purpose of the current study was to determine the independent contributions of these two temporal intervals on reinforcer accumulation in rats. Each left lever press earned a single food pellet; delivery of the accumulated pellet(s) occurred upon a right lever press. Conditions varied based on the presence of either an intertrial interval (ITI) that separated pellet delivery from the further opportunity to accumulate more pellets, or a delay-to-reinforcement that separated the right lever press from the delivery of the accumulated pellet(s). Delay and ITI values of 0, 5, 10 and 20 s were investigated. The delay-to-reinforcement conditions produced greater accumulation relative to the ITI conditions, despite accumulation increasing the density of reinforcement more substantially in the ITI conditions. This finding suggests that the temporal separation between reinforcer accumulation and subsequent delivery and consumption was a more critical variable in controlling reinforcer accumulation.

Substitution effects in a generalized token economy with pigeons

Pigeons made repeated choices between earning and exchanging reinforcer-specific tokens (green tokens exchangeable for food, red tokens exchangeable for water) and reinforcer-general tokens (white tokens exchangeable for food or water) in a closed token economy. Food and green food tokens could be earned on one panel; water and red water tokens could be earned on a second panel; white generalized tokens could be earned on either panel. Responses on one key produced tokens according to a fixed-ratio schedule, whereas responses on a second key produced exchange periods, during which all previously earned tokens could be exchanged for the appropriate commodity. Most conditions were conducted in a closed economy, and pigeons distributed their token allocation in ways that permitted food and water consumption. When the price of all tokens was equal and low, most pigeons preferred the generalized tokens. When token-production prices were manipulated, pigeons reduced production of the tokens that increased in price while increasing production of the generalized tokens that remained at a fixed price. The latter is consistent with a substitution effect: Generalized tokens increased and were exchanged for the more expensive reinforcer. When food and water were made freely available outside the session, token production and exchange was sharply reduced but was not eliminated, even in conditions when it no longer produced tokens. The results join with other recent data in showing sustained generalized functions of token reinforcers, and demonstrate the utility of token-economic methods for assessing demand for and substitution among multiple commodities in a laboratory context.

The several roles of stimuli in token reinforcement

Three experiments were conducted with pigeons to identify the stimulus functions of tokens in second-order token-reinforcement schedules. All experiments employed two-component multiple schedules with a token-reinforcement schedule in one component and a schedule with equivalent response requirements and/or reinforcer density in the other. In Experiment 1, response rates were lower under a token-reinforcement schedule than under a tandem schedule with the same response requirements, suggesting a discriminative role for the tokens. In Experiment 2, response rates varied systematically with signaling functions of the tokens in a series of conditions designed to explore other aspects of the temporal-correlative relations between tokens and food. In Experiment 3, response rates were reduced but not eliminated by presenting tokens independent of responding, yoked to their temporal occurrence in a preceding token component, suggesting both a reinforcing function and eliciting/evocative functions based on stimulus-food relations. Only when tokens were removed entirely was responding eliminated. On the whole, the results suggest that tokens, as stimuli temporally correlated with food, may serve multiple stimulus functions in token-reinforcement procedures--reinforcing, discriminative, or eliciting--depending on the precise arrangement of the contingencies in which they are embedded.

Rats' choices with token stimuli in concurrent variable-interval schedules

Four rats responded on concurrent variable-interval schedules that delivered token stimuli (stimulus lights arranged vertically above each of two side levers). During exchange periods, each token could be exchanged for one food pellet by responding on a center lever, with one response required for each pellet delivery. In different conditions, the exchange requirements (number of tokens that had to be earned before they could be exchanged for food) varied between one and four for the two response levers. The experiments were closely patterned after research with pigeons by Mazur and Biondi (2013), and the results from the rats in the present experiment were similar. Response percentages on the two levers changed as each additional token was earned, and these patterns indicated that choice was controlled by both the time to the exchange periods and the number of food pellets that were delivered in the exchange period. In some conditions, the exchange requirement was three tokens for each lever, but the token lights were not turned on as they were earned for one of the two keys. The rats showed a slight preference for the lever without the token lights, which may indicate that the token lights were not serving as conditioned reinforcers (a result also found by Mazur and Biondi with pigeons). Overall, these results suggest that, in this choice procedure, the token stimuli served primarily as discriminative stimuli that signaled the temporal proximity and quantity of the primary reinforcer, food.

Generalized conditioned reinforcement with pigeons in a token economy

Six pigeons were studied in a token economy in which tokens could be produced and exchanged for food on one side of an experimental chamber and for water on the opposite side of the chamber. Responses on one key produced tokens according to a token-production fixed ratio (FR) schedule. Responses on a second key produced an exchange period during which tokens were exchanged for water or food. In Experiment 1a, food tokens could be earned and exchanged under restricted food budgets, and water tokens could be earned and exchanged under water restricted budgets. In Experiment 1b, a third (generalized) token type could be earned and exchanged for either food or water under water restricted budgets. Across Experiments 1a and 1b, the number of tokens accumulated prior to exchange increased as the exchange-production schedule was increased. In Experiment 1b, pigeons produced more generalized than specific tokens, suggesting enhanced reinforcing efficacy of generalized tokens. In Experiment 2, the FR token-production price was manipulated under water restriction and then under food restriction. Production of each token type generally declined as a function of its own price and increased as a function of the price of the alternate type, demonstrating own-price and cross-price elasticity. Production of food and water tokens often changed together, indicating complementarity. Production of specific and generalized tokens changed in opposite directions, indicating substitutability. This is the first demonstration of sustained generalized functions of tokens in nonhumans, and illustrates a promising method for exploring economic contingencies in a controlled environment.

Distributed and accumulated reinforcement arrangements: evaluations of efficacy and preference

We assessed the efficacy of, and preference for, accumulated access to reinforcers, which allows uninterrupted engagement with the reinforcers but imposes an inherent delay required to first complete the task. Experiment 1 compared rates of task completion in 4 individuals who had been diagnosed with intellectual disabilities when reinforcement was distributed (i.e., 30-s access to the reinforcer delivered immediately after each response) and accumulated (i.e., 5-min access to the reinforcer after completion of multiple consecutive responses). Accumulated reinforcement produced response rates that equaled or exceeded rates during distributed reinforcement for 3 participants. Experiment 2 used a concurrent-chains schedule to examine preferences for each arrangement. All participants preferred delayed, accumulated access when the reinforcer was an activity. Three participants also preferred accumulated access to edible reinforcers. The collective results suggest that, despite the inherent delay, accumulated reinforcement is just as effective and is often preferred by learners over distributed reinforcement.

Risky choice in pigeons: preference for amount variability using a token-reinforcement system

Pigeons were given repeated choices between variable and fixed numbers of token reinforcers (stimulus lamps arrayed above the response keys), with each earned token exchangeable for food. The number of tokens provided by the fixed-amount option remained constant within blocks of sessions, but varied parametrically across phases, assuming values of 2, 4, 6, or 8 tokens per choice. The number of tokens provided by the variable-amount option varied between 0 and 12 tokens per choice, arranged according to an exponential or rectangular distribution. In general, the pigeons strongly preferred the variable option when the fixed option provided equal or greater numbers of tokens than the variable amount. Preference for the variable amount decreased only when the alternatives provided widely disparate amounts favoring the fixed amount. When tokens were removed from the experimental context, preference for the variable option was reduced or eliminated, suggesting that the token presentation played a key role in maintaining risk-prone choice patterns. Choice latencies varied inversely with preferences, suggesting that local analyses may provide useful ancillary measures of reinforcer value. Overall, the results indicate that systematic risk sensitivity can be attained with respect to reinforcer amount, and that tokens may be critical in the development of such preferences.

Pigeons' choices with token stimuli in concurrent variable-interval schedules

Twelve pigeons responded on concurrent variable-interval schedules that delivered token stimuli (stimulus lights for some pigeons, and white circles on the response keys for others). During exchange periods, each token could be exchanged for food on a fixed-ratio 1 schedule. Across conditions, the exchange requirements (number of tokens that had to be earned before they could be exchanged for food) varied between one and four for the two response keys. The main findings were that the pigeons' response percentages varied as a function of the number of tokens earned at any given moment, and they were determined by both the delays to food and by the number of food deliveries in the exchange periods. In some conditions, tokens had to be earned but were not visible during the variable-interval schedules for one or both keys. When one key had visible tokens and the other did not, the pigeons showed a preference for the key without visible tokens. A model based on the matching law and a hyperbolic delay-discounting equation could account for the main patterns of choice responding, and for how response percentages changed as successive tokens were earned. The results are consistent with the view that the token stimuli served as discriminative stimuli that signaled the current delays to food.

Rats (Rattus norvegicus) and pigeons (Columbia livia) are sensitive to the distance to food, but only rats request more food when distance increases

Three experiments investigated foraging by rats and pigeons. In Experiment 1, each response on a manipulandum delivered food to a cup, with the distance between the manipulandum and the cup varying across conditions. The number of responses made before traveling to collect and eat the food increased with distance for rats, but not for pigeons. In Experiment 2, two manipulanda were placed at different distances from a fixed food source; both pigeons and rats preferentially used the manipulandum closest to the food source. Experiment 3 was a systematic replication of Experiment 1 with pigeons. In different conditions, each peck on the left key increased the upcoming hopper duration by 0.5, 1.5 or 2.5s. Completing a ratio requirement on the right key of 1, 4, 8, 16 or 32 pecks, depending on the condition, then produced the food hopper for a duration that depended on the number of prior left pecks. As the ratio requirement increased on the right key, pigeons responded more on the left key and earned more food. Overall, the results replicate previous research, underlining similarities and differences between these species. The results are discussed in terms of optimal foraging, reinforcer sensitivity and delay discounting.

Token reinforcement: a review and analysis

Token reinforcement procedures and concepts are reviewed and discussed in relation to general principles of behavior. The paper is divided into four main parts. Part I reviews and discusses previous research on token systems in relation to common behavioral functions--reinforcement, temporal organization, antecedent stimulus functions, and aversive control--emphasizing both the continuities with other contingencies and the distinctive features of token systems. Part II describes the role of token procedures in the symmetrical law of effect, the view that reinforcers (gains) and punishers (losses) can be measured in conceptually analogous terms. Part III considers the utility of token reinforcement procedures in cross-species analysis of behavior more generally, showing how token procedures can be used to bridge the methodological gulf separating research with humans from that with other animals. Part IV discusses the relevance of token systems to the field of behavioral economics. Token systems have the potential to significantly advance research and theory in behavioral economics, permitting both a more refined analysis of the costs and benefits underlying standard economic models, and a common currency more akin to human monetary systems. Some implications for applied research and for broader theoretical integration across disciplines will also be considered.

Pathological gamblers discount probabilistic rewards less steeply than matched controls

Nineteen treatment-seeking men meeting Diagnostic and Statistical Manual for Mental Disorders (4th ed.) criteria for pathological gambling and 19 demographically matched controls participated. Participants provided demographic information, information about their recent drug use and gambling activities, and biological samples (to confirm drug abstinence). They also completed the Eysenck Personality Questionnaire, the South Oaks Gambling Screen (SOGS), and 2 questionnaires designed to separately quantify probability and delay discounting. Pathological gamblers discounted probabilistic rewards significantly less steeply than matched controls. A significant correlation revealed that more shallow probability discounting was associated with higher SOGS scores. Across groups, there was no significant difference in delay discounting, although this difference approached significance when education and ethnicity were included as covariates. These findings, collected for the 1st time with pathological gamblers, are consistent with previous reports that problem-gambling college students discount probabilistic rewards less steeply than controls. The nature of the relation between probability discounting and severity of problem gambling is deserving of further study.