The effect of conditioned reinforcement rate on choice: a review

An evaluation of methods for studying the effects of conditioned reinforcement on human choice

Shahan et al. (2006) found that the relative rate of pigeons' pecking on two observing responses (i.e., responses that only produced an S+ or stimulus correlated with primary reinforcement) was well described by the relative rate of S+ delivery. Researchers have not evaluated the effects of S+ delivery rate in a concurrent observing response procedure with human subjects, so the necessary procedural modifications for studying the effects of conditioned reinforcement on human choice remain unclear. The purpose of the current study was to conduct an additive component analysis of modifications to the procedures of Shahan et al. (2006). We evaluated the additive effects of introducing response cost, a changeover response, and ordinal discriminative stimuli on correspondence with the results of Shahan et al. and the quality of fits of the generalized matching equation. When our procedures were most similar to those of Shahan et al., we observed low rates of observing and indifference between the two observing responses. For the group of subjects with whom all three additive components were included, we obtained the highest level of sensitivity to relative rate of S+ delivery, but the slope and R² of our fits of the generalized matching equation were still much lower than those obtained by Shahan et al. Potential reasons for these discrepancies, methods of resolving them, and implications for future research are discussed.

Applied behavior analysis and the zoo: Forthman and Ogden (1992) thirty years later

The field of applied behavior analysis has been directly involved in both research and applications of behavioral principles to improve the lives of captive zoo animals. Thirty years ago, Forthman and Ogden (1992) wrote one of the first papers documenting some of these efforts. Since that time, considerable work has been done using behavioral principles and procedures to guide zoo welfare efforts. The current paper reexamines and updates Forthman and Ogden's original points, with attention to the 5 categories they detailed: (a) promotion of species-typical behavior, (b) reintroduction and repatriation of endangered species, (c) animal handling, (d) pest control, and (e) animal performances. In addition, we outline 3 current and future directions for behavior analytic endeavors: (a) experimental analyses of behavior and the zoo, (b) applied behavior analysis and the zoo, and (c) single-case designs and the zoo. The goal is to provide a framework that can guide future behavioral research in zoos, as well as create applications based on these empirical evaluations.

The Effect of the Establishment of Conditioned Reinforcement for Reading Content on Second-Graders' Reading Achievement

There remains a gap in the current literature as to how to reliably measure and increase students' "voluntary reading," based on research suggesting a relation between reading amount and reading achievement. We tested the effect of the establishment of conditioned reinforcement for reading via a collaborative shared reading (CSR) conditioning procedure on eight 2nd-grade students with and without learning disabilities and developmental disorders. This conditioning procedure was composed of opportunities for reciprocal reading and collaboration on comprehension and vocabulary tasks related to the reading content, such that partners (teacher-participant or participant-participant) were required to work together. We utilized a combined small-n experimental-control simultaneous-treatment design with a single-case multiple-probe design nested within each small group in order to compare within- and between-group differences for participants in the CSR procedure with a teacher or peer. All participants for whom conditioned reinforcement for reading was established (n = 7) demonstrated gains in reading achievement after a maximum of nine sessions (412 min), with grade-level increases between 0.2 and 2.5 on measures of reading comprehension and between 0.3 and 3.1 on measures of vocabulary. The students in the teacher-yoked condition (n = 3) demonstrated more significant gains in their average increases in achievement, although the peer-yoked procedure was also effective and possibly more viable in a classroom setting. These results suggest that a CSR procedure with a teacher or peer should be considered as a means of increasing the reading achievement of early elementary students via increases in the reinforcement value of reading.

Preference reversal between impulsive and self-control choice

In a concurrent-chains procedure, pigeons chose between reinforcers varying in delay and amount. Reinforcer amount was determined by duration of access to grain, and delay was determined by fixed-interval schedules in the terminal links. Preference was measured by the ratio of responses in initial links. Dependent scheduling of variable-interval schedules in initial links ensured that delay and amount were not confounded with frequency of outcomes, which remained equal for the two choices. In Experiment 1, in components signaled by red keys in the initial links, small and large reinforcers were delivered after delays of 1 s and 10 s respectively. In components signaled by green, additional time was added to both delays. Smaller-sooner reinforcers were preferred in red components. In green components, smaller-sooner reinforcers were preferred at short delays, and choices for the larger-later reinforcer generally increased with increasing duration of the added delay. At longer delays, up to 15 s, the larger-later reinforcer was preferred. That is, the pigeons showed within-session preference reversal, with impulsive choice at short delays in red components and self-control choice at long delays in green components. In Experiment 2, added delay to both reinforcement and reinforcer amount were varied. Sensitivity of initial-link response ratios to ratios of amount increased with increasing duration of the added delay. This interaction between delay and amount was predicted if the temporal discounting functions assumed the magnitude effect in which discounting rate was inversely proportional to amount. It was also predicted by the contextual choice model of performance in concurrent-chains procedures.

Perhaps More Consideration of Pavlovian-Operant Interaction May Improve the Clinical Efficacy of Behaviorally Based Drug Treatment Programs

Drug abuse remains costly. Drug-related cues can evoke cue-reactivity and craving, contributing to relapse. The Pavlovian extinction-based cue-exposure therapy (CET) has not been very successful in treating drug abuse. A functional operant analysis of complex rituals involved in CET is outlined and reinterpreted as an operant heterogeneous chain maintained by observing responses, conditioned reinforcers, and discriminative stimuli. It is further noted that operant functions are not predicated on Pavlovian processes but can be influenced by them in contributing to relapse; several empirical studies from the animal and human literature highlight this view. Cue-reactivity evoked by Pavlovian processes is conceptualized as an operant establishing/motivating operation. CET may be more effective in incorporating an operant-based approach that takes into account the complexity of Pavlovian-operant interaction. Extinction of the operant chain coupled with the shaping of alternative behaviors is proposed as an integrated therapy. It is proposed that operant-based drug abuse treatments (contingency management, voucher programs, and the therapeutic work environment) might consider incorporating cue-reactivity, as establishing/motivating operations, to increase long-term success-a hybrid approach based on Pavlovian-operant interaction.

Facets of Pavlovian and operant extinction

Research on extinction is of fundamental importance in both Pavlovian and operant approaches to the experimental analysis of learning. Although these approaches are often motivated by different empirical and theoretical questions, extinction has emerged as a research area in which common themes unite the two approaches. In this review, we focus on some common considerations in the analysis of Pavlovian and operant extinction. These include methodological challenges and interpretational issues in analyzing behavior during and after extinction. We consider the different roles that theory has played in the development of research on extinction in these preparations and conclude with some attention to applications of extinction.

A comparison of two pairing procedures to establish praise as a reinforcer

Some individuals with intellectual disabilities do not respond to praise as a reinforcer, which may limit their ability to learn. We evaluated 2 procedures (stimulus pairing and response-stimulus pairing), both of which involved pairing previously neutral praise statements with preferred edible items, to determine their usefulness in establishing praise as a reinforcer. Results of Study 1 indicated that stimulus pairing was not effective in conditioning praise as a reinforcer for 3 of 4 subjects; results were inconclusive for the 4th subject. Results of Study 2 indicated that response-stimulus pairing was effective in conditioning praise as a reinforcer for 4 of 8 subjects. After conditioning, praise also increased the occurrence of additional target responses for these 4 subjects.

Choice, conditioned reinforcement, and the prius effect

Psychologists have long been intrigued with the rationales that underlie our decisions. Similarly, the concept of conditioned reinforcement has a venerable history, particularly in accounting for behavior not obviously maintained by primary reinforcers. The studies of choice and of conditioned reinforcement have often developed in lockstep. Many contemporary approaches to these fundamental topics share an emphasis on context and on relative value. We trace the evolution of thinking about the potency of conditioned reinforcers from stimuli that were thought to acquire their value from pairings with more fundamental reinforcers to stimuli that acquire their value by being differentially correlated with these more fundamental reinforcers. We discuss some seminal experiments (including several that have been underappreciated) and some ongoing data, all of which have propelled us to the conclusion that the strength of conditioned reinforcers is determined by their signaling a relative improvement in the organism's relation to reinforcement.

Dual effects on choice of conditioned reinforcement frequency and conditioned reinforcement value

Pigeons were presented with a concurrent-chains schedule in which the total time to primary reinforcement was equated for the two alternatives (VI 30 s VI 60 s vs. VI 60 s VI 30 s). In one set of conditions, the terminal links were signaled by the same stimulus, and in another set of conditions they were signaled by different stimuli. Choice was in favor of the shorter terminal link when the terminal links were differentially signaled but in favor of the shorter initial link (and longer terminal link) when the terminal links shared the same stimulus. Preference reversed regularly with reversals of the stimulus condition and was unrelated to the discrimination between the two terminal links during the nondifferential stimulus condition. The present results suggest that the relative value of the terminal-link stimuli and the relative rate of conditioned reinforcer presentation are important influences on choice behavior, and that models of conditioned reinforcement need to include both factors.

A decision model for steady-state choice in concurrent chains

Grace and McLean (2006) proposed a decision model for acquisition of choice in concurrent chains which assumes that after reinforcement in a terminal link, subjects make a discrimination whether the preceding reinforcer delay was short or long relative to a criterion. Their model was subsequently extended by Christensen and Grace (2008, 2009a, 2009b) to include effects of initial- and terminal-link duration on choice. We show that an expression for steady-state responding can be derived from the decision model, which enables a model for choice that provides an account of archival data that is equal or superior to the contextual choice model (Grace, 1994) and hyperbolic value-added model (Mazur, 2001) in terms of goodness of fit, parsimony, and parameter invariance. The success of the steady-state decision model validates the strategy of understanding acquisition phenomena as a bridge toward explaining choice at the molar level.

Quantification of ethanol's antipunishment effect in humans using the generalized matching equation

Increases in rates of punished behavior by the administration of anxiolytic drugs (called antipunishment effects) are well established in animals but not humans. The present study examined antipunishment effects of ethanol in humans using a choice procedure. The behavior of 5 participants was placed under six concurrent variable-interval schedules of monetary reinforcement. In three of the six concurrent schedules, punishment, in the form of monetary loss, was superimposed on one alternative. Data were analyzed according to the generalized matching equation which distinguishes between bias (allocation of behavior beyond what matching to relative reinforcer densities would predict) and sensitivity to reinforcement (how well behavior tracks relative reinforcer densities). In addition, participants completed a pencil-tapping test. Under placebo punishment conditions, all participants demonstrated low response rates and a bias against the alternative associated with punishment, despite a resultant loss of available reinforcers. Bias against the punished alternative was dose-dependently reduced in participants shown to be most sensitive to ethanol (0.6, 1.2, and 1.8 g/kg) in measures of overall responding and on the pencil-tapping test. No ethanol-induced change in bias was noted when punishment was not imposed. Sensitivity to reinforcement also decreased for participants shown to be sensitive to ethanol. In addition to extending antipunishment effects to humans, these results also show that antipunishment effects can be quantified via the matching equation.

Conditioned reinforcement and response strength

Stimuli associated with primary reinforcers appear themselves to acquire the capacity to strengthen behavior. This paper reviews research on the strengthening effects of conditioned reinforcers within the context of contemporary quantitative choice theories and behavioral momentum theory. Based partially on the finding that variations in parameters of conditioned reinforcement appear not to affect response strength as measured by resistance to change, long-standing assertions that conditioned reinforcers do not strengthen behavior in a reinforcement-like fashion are considered. A signposts or means-to-an-end account is explored and appears to provide a plausible alternative interpretation of the effects of stimuli associated with primary reinforcers. Related suggestions that primary reinforcers also might not have their effects via a strengthening process are explored and found to be worthy of serious consideration.

Influence of conditioned reinforcement on the response-maintaining effects of quinpirole in rats

D2-like agonists, such as quinpirole, maintain responding in monkeys, rats, and mice when they are substituted for cocaine. This study examined the influence of operant history and cocaine-paired stimuli (CS) on quinpirole-maintained responding in rats trained to nose poke for cocaine. Upon acquisition of responding for cocaine, substitutions were performed in the presence or absence of injection-CS pairings. Although cocaine maintained responding regardless of whether injections were accompanied by CS, quinpirole maintained responding only when CS were paired with injections. To assess the influence of operant history, injections of cocaine, quinpirole, remifentanil, nicotine, or saline were made available on a previously inactive lever, while nose pokes continued to result in CS presentation. Although responding was reallocated from the nose poke to the lever when cocaine or remifentanil was available, lever presses remained low, and nose poking persisted when quinpirole or nicotine was made contingent upon lever presses. Finally, quinpirole pretreatments resulted in high rates of nose poking when nose pokes resulted in CS presentation alone, but failed to maintain nose poking when the CS was omitted. Taken together, these results suggest that the response-maintaining effects of quinpirole are primarily mediated by an enhancement of the conditioned reinforcing effects of earlier CS, rather than by a reinforcing effect of quinpirole.

Response allocation in concurrent chains when terminal-link delays follow an ascending and descending series

Eight pigeons were trained in a concurrent-chains procedure in which the terminal-link immediacy ratio followed an ascending or descending series. Across sessions, one terminal-link delay changed from 2 s to 32 s to 2 s or from 32 s to 2 s to 32 s, while the other was always 8 s. For all pigeons, response allocation tracked changes in delay and was biased towards the 8-s alternative on the descending series, indicating a hysteresis effect, and was more sensitive to changes in the terminal-link delay ratio for relatively long (> 8 s) than short (< 8 s) delays. Both the hysteresis and effect of delay duration were predicted by an extended version of Grace and McLean's (2006) decision model. The extended decision model provided an overall better account of the results than a simple linear-operator model (Grace, 2002), and holds promise for an integrated account of choice in concurrent chains for both acquisition and steady-state conditions.

The probability of small schedule values and preference for random-interval schedules

Preference for working on variable schedules and temporal discrimination were simultaneously examined in two experiments using a discrete-trial, concurrent-chains arrangement with fixed interval (FI) and random interval (RI) terminal links. The random schedule was generated by first sampling a probability distribution after the programmed delay to reinforcement on the FI schedule had elapsed, and thus the RI never produced a component schedule value shorter than the FI and maintained a rate of reinforcement half that of the FI. Despite these features, the FI was not strongly preferred. The probability of obtaining the smallest programmed delay to reinforcement on the RI schedule was manipulated in Experiment 1, and the interaction of this probability and initial link length was examined in Experiment 2. As the probability of obtaining small values in the RI increased, preference for the schedule increased while the discriminated time of reinforcer availability in the terminal link decreased. Both of these effects were attenuated by lengthening the initial links. The results support the view that in addition to the delay to reinforcement, the probability of obtaining a short delay is an important choice-affecting variable that likely contributes to the robust preferences for variable, as opposed to fixed, schedules of reinforcement.

Conditioned reinforcement value and resistance to change

Three experiments examined the effects of conditioned reinforcement value and primary reinforcement rate on resistance to change using a multiple schedule of observing-response procedures with pigeons. In the absence of observing responses in both components, unsignaled periods of variable-interval (VI) schedule food reinforcement alternated with extinction. Observing responses in both components intermittently produced 15 s of a stimulus associated with the VI schedule (i.e., S+). In the first experiment, a lower-valued conditioned reinforcer and a higher rate of primary reinforcement were arranged in one component by adding response-independent food deliveries uncorrelated with S+. In the second experiment, one component arranged a lower valued conditioned reinforcer but a higher rate of primary reinforcement by increasing the probability of VI schedule periods relative to extinction periods. In the third experiment, the two observing-response components provided similar rates of primary reinforcement but arranged different valued conditioned reinforcers. Across the three experiments, observing-response rates were typically higher in the component associated with the higher valued conditioned reinforcer. Resistance to change was not affected by conditioned reinforcement value, but was an orderly function of the rate of primary reinforcement obtained in the two components. One interpretation of these results is that S+ value does not affect response strength and that S+ deliveries increase response rates through a mechanism other than reinforcement. Alternatively, because resistance to change depends on the discriminative stimulus-reinforcer relation, the failure of S+ value to impact resistance to change could have resulted from a lack of transfer of S+ value to the broader discriminative context.

Failure to obtain value enhancement by within-trial contrast in simultaneous and successive discriminations

The present research tested the generality of the "work ethic" effect described by Clement, Feltus, Kaiser, and Zentall (2000). In Experiment 1, we trained 10 pigeons on a pair of either simultaneous or successive discriminations. One discrimination followed a high-effort requirement (20 pecks to the center key) and the other followed a low-effort requirement (1 peck). Contrary to Clement et al.'s results, we found that preferences between the S+ and S- stimuli in transfer tests depended on the event that initiated the trial: Pigeons preferred the stimulus from the baseline discrimination whose initiating event was most dissimilar from that preceding the test trial. Preferences were similar but less extreme in the successive condition. In Experiment 2, we investigated whether test preferences depended on the amount of training. A total of 12 pigeons were trained on a pair of simultaneous discriminations, except that test sessions were scheduled after every three baseline sessions. Preferences increased across test sessions but were similar to those in Experiment 1. Together with Vasconcelos, Urcuioli, and Lionello-DeNolf (2007a), our study represents a second failure to replicate Clement et al.'s work ethic effect. The finding that preference depends on the event that initiates the test trial suggests that choice probes may not provide unambiguous assessments of stimulus value.

Signals, resistance to change, and conditioned reinforcement in a multiple schedule

The effect of signals on resistance to change was evaluated using pigeons responding on a three-component multiple schedule. Each component contained a variable-interval initial link followed by a fixed-time terminal link. One component was an unsignaled-delay schedule, and two were equivalent signaled-delay schedules. After baseline training, resistance to change was assessed through (a) extinction and (b) adding free food to the intercomponent interval. During these tests, the signal stimulus from one of the signaled-delay components (SIG-T) was replaced with the initial-link stimulus from that component, converting it to an unsignaled-delay schedule. That signal stimulus was added to the delay period of the unsignaled-delay component (UNS), converting it to a signaled-delay schedule. The remaining signaled component remained unchanged (SIG-C). Resistance-to-change tests showed removing the signal had a minimal effect on resistance to change in the SIG-T component compared to the unchanged SIG-C component except for one block during free-food testing. Adding the signal to the UNS component significantly increased response rates suggesting that component had low response strength. Interestingly, the direction of the effect was in the opposite direction from what is typically observed. Results are consistent with the conclusion that the signal functioned as a conditioned reinforcer and inconsistent with a generalization-decrement explanation.

The Journal of the Experimental Analysis of Behavior at zero, fifty, and one hundred

The experimental content areas represented in JEAB in its first volume (1958) and 50 years later in Volume 87 are in many ways similar with regard to research on schedules of reinforcement, research with human subjects, and several other topics. Experimental analysis has not been displaced by quantitative analysis. Much less research on aversive control has been published in recent than in earlier years. Wishes for progress in the next 50 years include experiments on verbal behavior, the sources of novel behavior, and observing responses based on stimuli correlated with escape or avoidance.

Quantitative analyses of observing and attending

We review recent experiments examining whether simple models of the allocation and persistence of operant behavior are applicable to attending. In one series of experiments, observing responses of pigeons were used as an analog of attending. Maintenance of observing is often attributed to the conditioned reinforcing effects of a food-correlated stimulus (i.e., S+), so these experiments also may inform our understanding of conditioned reinforcement. Rates and allocations of observing were governed by rates of food or S+ delivery in a manner consistent with the matching law. Resistance to change of observing was well described by behavioral momentum theory only when rates of primary reinforcement in the context were considered. Rate and value of S+ deliveries did not affect resistance to change. Thus, persistence of attending to stimuli appears to be governed by primary reinforcement rates in the training context rather than conditioned reinforcing effects of the stimuli. An additional implication of these findings is that conditioned "reinforcers" may affect response rates through some mechanism other than response-strengthening. In a second series of experiments, we examined the applicability of the matching law to the allocation of attending to the elements of compound stimuli in a divided-attention task. The generalized matching law described performance well, and sensitivity to relative reinforcement varied with sample duration. The bias and sensitivity terms of the generalized matching law may provide measures of stimulus-driven and goal-driven control of divided attention. Further application of theories of operant behavior to performance on attention tasks may provide insights into what is referred to variously as endogenous, top-down, or goal-directed control of attention.

Effect of unsignaled delays between stimuli in a chain schedule on responding and resistance to change

Behavioral momentum theory is an evolving theoretical account of the strength of behavior. One challenge for the theory is determining the role of signal stimuli in determining response strength. This study evaluated the effect of an unsignaled delay between the initial link and terminal link of a two-link chain schedule on resistance to change using a multiple schedule of reinforcement. Pigeons were presented two different signaled delay to reinforcement schedules. Both schedules employed a two-link chain schedule with a variable interval 120-s initial link followed by a 5-s fixed time terminal link schedule. One of the schedules included a 5-s unsignaled delay between the initial link and the terminal link. Resistance to change was assessed with two separate disruption procedures: extinction and adding a variable time 20-s schedule of reinforcement to the inter-component interval. Baseline responding was lower in the schedule with the unsignaled delay but resistance to change for the initial link was unaffected by the unsignaled delay. The results suggest that not all unsignaled delays are equal in their effect on resistance to change.