Second-order schedules of token reinforcement with pigeons: effects of fixed- and variable-ratio exchange schedules

Pavlovian learning and conditioned reinforcement

Conditioned reinforcers are widely used in applied behavior analysis. Basic research evidence reveals that Pavlovian learning plays an important role in the acquisition and efficacy of new conditioned-reinforcer functions. Thus, a better understanding of Pavlovian principles holds the promise of improving the efficacy of conditioned reinforcement in applied research and practice. This paper surveys how (and if) Pavlovian principles are presented in behavior-analytic textbooks; imprecisions and knowledge gaps within contemporary Pavlovian empirical findings are highlighted. Thereafter, six practical principles of Pavlovian conditioning are presented along with empirical support and knowledge gaps that should be filled by applied and translational behavior-analytic researchers. Innovative applications of these principles are outlined for research in language acquisition, token reinforcement, and self-control.

Cooperative responding in rats: II. Performance on fixed-ratio schedules of mutual reinforcement

Coordinated responses of 5 dyads of rats were investigated under fixed-ratio (FR) schedules of mutual water reinforcement. Coordinated responding was defined as 2 consecutive lever-presses, 1 from each of 2 rats, occurring <.5 s apart. In the FR schedules, each coordinated episode was defined as 1 response in the FR sequence. The size of FR schedules was parametrically manipulated assuming the values of FR 1, 6, 12, 18, 24, 30, 50, and 9, in this order. Each FR remained in effect until responding reached stability. Under all conditions, pairs of rats received access to water simultaneously (mutual reinforcement). Rates and proportions of coordinated responding showed a bitonic inverted U-shaped function of ratio size. Postreinforcement pauses increased systematically as the interreinforcement interval increased. Local rates and proportions increased as a function of response location within ratios. Results of a control condition with relaxed temporal constraints for mutual reinforcement showed decreases in rates and proportion of coordinated responses, suggesting that the coordinated responses were controlled by the mutual reinforcement contingencies. The present experiment showed that coordinated responding is quantitatively affected by 3 properties of FR schedules: response requirement, reinforcement rates, and proximity to reinforcement.

Token reinforcement: Translational research and application

The present paper provides an integrative review of research on token reinforcement systems, organized in relation to basic behavioral functions and economic variables. This type of functional taxonomy provides a useful way to organize the literature, bringing order to a wide range of findings across species and settings, and revealing gaps in the research and areas especially ripe for analysis and application. Unlike standard translational research, based on a unidirectional model in which the analysis moves from laboratory to the applied realm, work in the area of token systems is best served by a bidirectional interplay between laboratory and applied research, where applied questions inspire research on basic mechanisms. When based on and contributing to an analysis, applied research on token economies can be on the leading edge of theoretical advances, helping set the scientific research agenda.

The several roles of stimuli in token reinforcement

Three experiments were conducted with pigeons to identify the stimulus functions of tokens in second-order token-reinforcement schedules. All experiments employed two-component multiple schedules with a token-reinforcement schedule in one component and a schedule with equivalent response requirements and/or reinforcer density in the other. In Experiment 1, response rates were lower under a token-reinforcement schedule than under a tandem schedule with the same response requirements, suggesting a discriminative role for the tokens. In Experiment 2, response rates varied systematically with signaling functions of the tokens in a series of conditions designed to explore other aspects of the temporal-correlative relations between tokens and food. In Experiment 3, response rates were reduced but not eliminated by presenting tokens independent of responding, yoked to their temporal occurrence in a preceding token component, suggesting both a reinforcing function and eliciting/evocative functions based on stimulus-food relations. Only when tokens were removed entirely was responding eliminated. On the whole, the results suggest that tokens, as stimuli temporally correlated with food, may serve multiple stimulus functions in token-reinforcement procedures--reinforcing, discriminative, or eliciting--depending on the precise arrangement of the contingencies in which they are embedded.

Rats' choices with token stimuli in concurrent variable-interval schedules

Four rats responded on concurrent variable-interval schedules that delivered token stimuli (stimulus lights arranged vertically above each of two side levers). During exchange periods, each token could be exchanged for one food pellet by responding on a center lever, with one response required for each pellet delivery. In different conditions, the exchange requirements (number of tokens that had to be earned before they could be exchanged for food) varied between one and four for the two response levers. The experiments were closely patterned after research with pigeons by Mazur and Biondi (2013), and the results from the rats in the present experiment were similar. Response percentages on the two levers changed as each additional token was earned, and these patterns indicated that choice was controlled by both the time to the exchange periods and the number of food pellets that were delivered in the exchange period. In some conditions, the exchange requirement was three tokens for each lever, but the token lights were not turned on as they were earned for one of the two keys. The rats showed a slight preference for the lever without the token lights, which may indicate that the token lights were not serving as conditioned reinforcers (a result also found by Mazur and Biondi with pigeons). Overall, these results suggest that, in this choice procedure, the token stimuli served primarily as discriminative stimuli that signaled the temporal proximity and quantity of the primary reinforcer, food.

Generalized conditioned reinforcement with pigeons in a token economy

Six pigeons were studied in a token economy in which tokens could be produced and exchanged for food on one side of an experimental chamber and for water on the opposite side of the chamber. Responses on one key produced tokens according to a token-production fixed ratio (FR) schedule. Responses on a second key produced an exchange period during which tokens were exchanged for water or food. In Experiment 1a, food tokens could be earned and exchanged under restricted food budgets, and water tokens could be earned and exchanged under water restricted budgets. In Experiment 1b, a third (generalized) token type could be earned and exchanged for either food or water under water restricted budgets. Across Experiments 1a and 1b, the number of tokens accumulated prior to exchange increased as the exchange-production schedule was increased. In Experiment 1b, pigeons produced more generalized than specific tokens, suggesting enhanced reinforcing efficacy of generalized tokens. In Experiment 2, the FR token-production price was manipulated under water restriction and then under food restriction. Production of each token type generally declined as a function of its own price and increased as a function of the price of the alternate type, demonstrating own-price and cross-price elasticity. Production of food and water tokens often changed together, indicating complementarity. Production of specific and generalized tokens changed in opposite directions, indicating substitutability. This is the first demonstration of sustained generalized functions of tokens in nonhumans, and illustrates a promising method for exploring economic contingencies in a controlled environment.

Risky choice in pigeons: preference for amount variability using a token-reinforcement system

Pigeons were given repeated choices between variable and fixed numbers of token reinforcers (stimulus lamps arrayed above the response keys), with each earned token exchangeable for food. The number of tokens provided by the fixed-amount option remained constant within blocks of sessions, but varied parametrically across phases, assuming values of 2, 4, 6, or 8 tokens per choice. The number of tokens provided by the variable-amount option varied between 0 and 12 tokens per choice, arranged according to an exponential or rectangular distribution. In general, the pigeons strongly preferred the variable option when the fixed option provided equal or greater numbers of tokens than the variable amount. Preference for the variable amount decreased only when the alternatives provided widely disparate amounts favoring the fixed amount. When tokens were removed from the experimental context, preference for the variable option was reduced or eliminated, suggesting that the token presentation played a key role in maintaining risk-prone choice patterns. Choice latencies varied inversely with preferences, suggesting that local analyses may provide useful ancillary measures of reinforcer value. Overall, the results indicate that systematic risk sensitivity can be attained with respect to reinforcer amount, and that tokens may be critical in the development of such preferences.

Token reinforcement: a review and analysis

Token reinforcement procedures and concepts are reviewed and discussed in relation to general principles of behavior. The paper is divided into four main parts. Part I reviews and discusses previous research on token systems in relation to common behavioral functions--reinforcement, temporal organization, antecedent stimulus functions, and aversive control--emphasizing both the continuities with other contingencies and the distinctive features of token systems. Part II describes the role of token procedures in the symmetrical law of effect, the view that reinforcers (gains) and punishers (losses) can be measured in conceptually analogous terms. Part III considers the utility of token reinforcement procedures in cross-species analysis of behavior more generally, showing how token procedures can be used to bridge the methodological gulf separating research with humans from that with other animals. Part IV discusses the relevance of token systems to the field of behavioral economics. Token systems have the potential to significantly advance research and theory in behavioral economics, permitting both a more refined analysis of the costs and benefits underlying standard economic models, and a common currency more akin to human monetary systems. Some implications for applied research and for broader theoretical integration across disciplines will also be considered.

Reinforcer accumulation in a token-reinforcement context with pigeons

Four pigeons were exposed to a token-reinforcement procedure with stimulus lights serving as tokens. Responses on one key (the token-production key) produced tokens that could be exchanged for food during an exchange period. Exchange periods could be produced by satisfying a ratio requirement on a second key (the exchange-production key). The exchange-production key was available any time after one token had been produced, permitting up to 12 tokens to accumulate prior to exchange. Token accumulation, measured in terms of both frequency (percent cycles with accumulation) and magnitude (mean number of tokens accumulated), decreased as the token-production ratio increased from 1 to 10 across conditions (with exchange-production ratio held constant), and increased as the exchange-production ratio increased from 1 to 250 across conditions (with token-production ratio held constant). When tokens were removed, accumulation decreased markedly compared to conditions with tokens and the same schedules. These data show that token accumulation is an orderly function of token-production and exchange-production schedules, and they are broadly consistent with a unit-price model based on local and global responses per reinforcer.

Second-order schedules of token reinforcement with pigeons: implications for unit price

Four pigeons were exposed to second-order schedules of token reinforcement, with stimulus lights serving as token reinforcers. Tokens were earned according to a fixed-ratio (token-production) schedule, with the opportunity to exchange tokens for food (exchange period) occurring after a fixed number had been produced (exchange-production ratio). The token-production and exchange-production ratios were manipulated systematically across conditions. Response rates varied inversely with the token-production ratio at each exchange-production ratio. Response rates also varied inversely with the exchange-production ratio at each token-production ratio, particularly at the higher token-production ratios. At higher token-production and exchange-production ratios, response rates increased in token-production segments closer to exchange periods and food. Some conditions were conducted in a closed economy, in which the pigeons earned all their daily ration of food within the session. Relative to comparable open-economy conditions, response rates in the closed economy were less affected by changes in token-production ratio, resulting in higher levels of food intake and body weight. Some of the results are consistent with the economic concept of unit price, a cost-benefit ratio comprised of responses per unit of food delivery, but most are well accounted for by a consideration of the number of responses required to produce exchange periods, without regard to the amount of reinforcement available during those exchange periods.

Response-cost punishment via token loss with pigeons

Two experiments were conducted to investigate punishment via response-contingent removal of conditioned token reinforcers (response cost) with pigeons. In Experiment 1, key pecking was maintained on a two-component multiple second-order schedule of token delivery, with light emitting diodes (LEDs) serving as token reinforcers. In both components, responding produced tokens according to a random-interval 20-s schedule and exchange periods according to a variable-ratio schedule. During exchange periods, each token was exchangeable for 2.5-s access to grain. In one component, responses were conjointly punished according to fixed-ratio schedules of token removal. Response rates in this punishment component decreased to low levels while response rates in the alternate (no-punishment) component were unaffected. Responding was eliminated when it produced neither tokens nor exchange periods (Extinction), but was maintained at moderate levels when it produced tokens in the signaled absence of food reinforcement, suggesting that tokens served as effective conditioned reinforcers. In Experiment 2, the effect of the response-cost punishment contingency was separated from changes in the density of food reinforcement. This was accomplished by yoking either the number of food deliveries per component (Yoked Food) or the temporal placement of all stimulus events (tokens, exchanges, food deliveries) (Yoked Complete), from the punishment to the no-punishment component. Response rates decreased in both components, but decreased more rapidly and were generally maintained at lower levels in the punishment component than in the yoked component. In showing that the response-cost contingency had a suppressive effect on responding in addition to that produced by reductions in reinforcement density, the present results suggest that response-cost punishment shares important features with other forms of punishment.

Unit price and choice in a token-reinforcement context

Pigeons were exposed to multiple and concurrent second-order schedules of token reinforcement, with stimulus lights serving as token reinforcers. Tokens were produced and exchanged for food according to various fixed-ratio schedules, yielding equal and unequal unit prices (responses per unit food delivery). On one schedule (termed the standard schedule), the unit price was held constant across conditions. On a second schedule (the alternative schedule), the unit price was either the same or different from the standard. Under conditions with unequal unit prices, near-exclusive preference for the lower unit price was obtained. Under conditions with equal unit prices, the direction and degree of preference depended on ratio size (number of responses per exchange period). When this ratio differed, strong preferences for the smaller ratio were observed. When this ratio was equal, preferences were nearer indifference. Response rates on the multiple schedule were generally consistent with the preference data in showing sensitivity to ratio size. Results are discussed in terms of a unit-price model that includes handling and reinforcer immediacy as additional costs. On the whole, results show that preferences were determined primarily by delay to the exchange period.

Determinants of pigeons' choices in token-based self-control procedures

Four pigeons were exposed to a token-based self-control procedure with stimulus lights serving as token reinforcers. Smaller-reinforcer choices produced one token immediately; larger-reinforcer choices produced three tokens following a delay. Each token could be exchanged for 2-s access to food during a signaled exchange period each trial. The main variables of interest were the exchange delays (delays from the choice to the exchange stimulus) and the food delays (also timed from the choice), which were varied separately and together across blocks of sessions. When exchange delays and food delays were shorter following smaller-reinforcer choices, strong preference for the smaller reinforcer was observed. When exchange delays and food delays were equal for both options, strong preference for the larger reinforcer was observed. When food delays were equal for both options but exchange delays were shorter for smaller-reinforcer choices, preference for the larger reinforcer generally was less extreme than under conditions in which both exchange and food delays were equal. When exchange delays were equal for both options but food delays were shorter for smaller-reinforcer choices, preference for the smaller reinforcer generally was less extreme than under conditions in which both exchange and food delays favored smaller-reinforcer choices. On the whole, the results were consistent with prior research on token-based self-control procedures in showing that choices are governed by reinforcer immediacy when exchange and food delays are unequal and by reinforcer amount when exchange and food delays are equal. Further, by decoupling the exchange delays from food delays, the results tentatively support a role for the exchange stimulus as a conditioned reinforcer.