A molecular analysis of choice on concurrent-chains schedules

A decision model for steady-state choice in concurrent chains

Grace and McLean (2006) proposed a decision model for acquisition of choice in concurrent chains which assumes that after reinforcement in a terminal link, subjects make a discrimination whether the preceding reinforcer delay was short or long relative to a criterion. Their model was subsequently extended by Christensen and Grace (2008, 2009a, 2009b) to include effects of initial- and terminal-link duration on choice. We show that an expression for steady-state responding can be derived from the decision model, which enables a model for choice that provides an account of archival data that is equal or superior to the contextual choice model (Grace, 1994) and hyperbolic value-added model (Mazur, 2001) in terms of goodness of fit, parsimony, and parameter invariance. The success of the steady-state decision model validates the strategy of understanding acquisition phenomena as a bridge toward explaining choice at the molar level.

Delay of reinforcers in a concurrent-chain schedule: An extension of the hyperbolic-decay model

Six pigeons were trained in concurrent-chain schedules with equal aperiodic initial links and delays to reinforcers in the terminal links. The terminal links always lasted 30 s. In Experiment 1, two reinforcers were delivered in each terminal link, with the first reinforcer delivered either 1 s (Experiment 1A) or 5 s (Experiment 1B) after choice. In these experiments, the delay between the first and second reinforcers in one terminal link was 10 s, and the delay between the first and second reinforcers on the other key was varied. This variation produced little change in preference. In Experiment 1C, the first and second delays on one key were 10 s, and on the other key they were varied within the restriction that the sum of delays was 20 s. Preference for the varied terminal link increased as the first delay was decreased. A hyperbolic model of the value of reinforcer delay provided a good description of the data from Experiment 1. In Experiment 2, a single reinforcer was delivered in each terminal link after a delay of either 0.2 or 19.8 s, and these delays were reversed between conditions. The initial-link schedule providing terminal-link access was varied from means of 5 s to 480 s. As the initial-link duration was increased, preference for the shorter delay became less extreme. An extension of the hyperbolic-decay model, in which the decay constant was a hyperbolic function of the initial-link duration, described the results well. Differences between the procedure used here (constant-duration terminal links) and that used in conventional concurrent-chain research precludes use of the model as a general account of concurrent-chain performance.

Concurrent-chain performance: Effects of absolute and relative terminal-link entry frequency

Six pigeons were trained in a concurrent-chain procedure with constant variable-interval 6-s variable-interval 12-s terminal links. Five groups of conditions were arranged. Within a group of conditions, the duration of one initial-link schedule was held constant and the duration of the other initial link was varied. The duration of the varied initial link was always longer than, or equal to, the constant initial-link duration. The duration of the shorter initial link was varied across groups of conditions from 5 s to 70 s. The data from each group were well described by the generalized matching law. Sensitivity (a) to the terminal-link entry ratio increased as the shorter initial-link duration increased, but appeared to reach an asymptote at shorter initial-link durations greater than 32 s. Terminal-link bias did not change with changes in shorter initial-link duration for the response-allocation data, but showed a small increase with increasing shorter initial-link duration for the time-allocation data.

Short-term and long-term effects of reinforcers on choice

The relation between molar and molecular aspects of time allocation was studied in pigeons on concurrent variable-time variable-time schedules of reinforcement. Fifteen-minute reinforcer-free periods were inserted in the middle of every third session. Generalized molar matching of time ratios to reinforcer ratios was observed during concurrent reinforcement. Contrary to melioration theory, preference was unchanged during the reinforcer-free periods as well as in extinction. In addition to this long-term effect of reinforcement, short-term effects were observed: Reinforcers increased the duration of the stays during which they were delivered but had little consistent effect either on the immediately following stay in the same schedule or on the immediately following stay in the alternative schedule. Thus, an orderly effect of reinforcer delivery on molecular aspects of time allocation was observed, but because of its short-term nature, this effect cannot account for the matching observed at the molar level.

Choice and delay of reinforcement: Effects of terminal-link stimulus and response conditions

In two experiments, pigeons were exposed to concurrent-chains schedules in which a single initial-link variable-interval schedule led to access to terminal links composed of fixed-interval or fixed-delay schedules. In Experiment 1, an 8-s (or 16-s) delay to reinforcement was associated with the standard key, while reinforcer delay values associated with the experimental key were varied from 4 to 32 s. The results of Experiment 1 showed undermatching of response ratios to delay ratios with terminal-link fixed-delay schedules, whereas in some pigeons matching or overmatching was evident with the fixed-interval schedules. In Experiment 2, one pair of reinforcer delay values, either 8 versus 16 s or 16 versus 32 s, was used. In the first condition of Experiment 2, different delays were associated with different keylight stimuli (cued condition). In the second condition, different terminal-link delays were associated with the same stimulus, either a blackout (uncued-blackout condition) or a white key (uncued-white condition). To examine the role of responses emitted during delays, the keys were retracted during a delay (key-absent condition) in the third condition and responses were required by a fixed-interval schedule in the fourth condition. Experiment 2 demonstrated that the choice proportions for the shorter delay were more extreme in the cued condition than in the uncued-blackout condition, and that the response requirement imposed by the fixed-interval schedules did not affect choice of the shorter delay, nor did the key-absent and key-present conditions. These results indicate that the keylight-stimulus conditions affected preference for the shorter of two delays and that the findings obtained in Experiment 1 depended mainly on the keylight-stimulus conditions of the terminal links (i.e., the conditioned reinforcing value of the terminal-link stimuli).

A contextual model of concurrent-chains choice

An extension of the generalized matching law incorporating context effects on terminal-link sensitivity is proposed as a quantitative model of behavior under concurrent chains. The contextual choice model makes many of the same qualitative predictions as the delay-reduction hypothesis, and assumes that the crucial contextual variable in concurrent chains is the ratio of average times spent, per reinforcement, in the terminal and initial links; this ratio controls differential effectiveness of terminal-link stimuli as conditioned reinforcers. Ninety-two concurrent-chains data sets from 19 published studies were fitted to the model. Averaged across all studies, the model accounted for 90% of the variance in pigeons' relative initial-link responding. The model therefore demonstrates that a matching law analysis of concurrent chains-the assumption that relative initial-link responding equals relative terminal-link value-remains quantitatively viable. Because the model reduces to the generalized matching law when terminal-link duration is zero, it provides a quantitative integration of concurrent schedules and concurrent chains.

Local effects of delayed food

Five pigeons were trained on a procedure in which seven concurrent variable-interval schedules arranged seven different food-rate ratios in random sequence in each session. Each of these components lasted for 10 response-produced food deliveries, and components were separated by 10-s blackouts. We varied delays to food (signaled by blackout) between the two response alternatives in an experiment with three phases: In Phase 1, the delay on one alternative was 0 s, and the other was varied between 0 and 8 s; in Phase 2, both delays were equal and were varied from 0 to 4 s; in Phase 3, the two delays summed to 8 s, and each was varied from 1 to 7 s. The results showed that increasing delay affected local choice, measured by a pulse in preference, in the same way as decreasing magnitude, but we found also that increasing the delay at the other alternative increased local preference. This result casts doubt on the traditional view that a reinforcer strengthens a response depending only on the reinforcer's value discounted by any response-reinforcer delay. The results suggest that food guides, rather than strengthens, behavior.

Choice and timing in concurrent chains: Effects of initial-link duration

Theories of timing have been applied to choice between delayed rewards by assuming that delays are represented in memory and that subjects sample from memory when choosing between alternatives. To search for covariation in single-trial measures of performance that might confirm this assumption, we used a procedure that allowed for convergent measurement of choice and timing behavior. Four pigeons responded in a concurrent chains/peak procedure in which the terminal links were fixed-interval (FI) 8s and FI 16s, across conditions the duration of the initial-link schedule was either short or long, and one quarter of the terminal links lasted for 48 s and ended without reinforcer delivery. Preference for the FI 8-s alternative was stronger with shorter initial links, replicating the 'initial-link effect'. Responding on no-food trials was unaffected by initial-link duration, and aggregated across trials, was typical of the peak procedure: response distributions were approximately Gaussian, with modes near the FI schedule durations, and variance was greater for the FI 16-s terminal link. Analysis of local measures of initial-link performance (e.g., pause to begin responding, time spent responding, number and duration of visits to each alternative, etc.) found that the initial-link effect was associated with an increase in the number and duration of visits per cycle to the nonpreferred alternative. Regression analyses showed that local initial-link measures contributed relatively little additional variance in predicting performance on individual no-food trials beyond that accounted for by FI schedule. Our results provide no clear evidence that initial- and terminal-link responding in concurrent chains are mediated by a common representation of terminal-link delays.

Temporal context in concurrent chains: I. Terminal-link duration

Two experiments are reported in which the ratio of the average times spent in the terminal and initial links (Tt/Ti) in concurrent chains was varied. In Experiment 1, pigeons responded in a three-component procedure in which terminal-link variable-interval schedules were in constant ratio, but their average duration increased across components by a factor of two. The log initial-link response ratio was a negatively accelerated function of Tt/Ti. Overall, the data were well described by Grace's (1994) contextual choice model (CCM) with temporal context represented as (Tt/Ti)k or 2Tt/(Tt + Ti), and by Mazur's (2001) hyperbolic value-added model (HVA), with each model accounting for approximately 93% of the variance. In Experiment 2, fixed-parameter predictions for each model were generated, based on the data from Experiment 1, for conditions in which Tt/Ti was varied over a more extreme range. Data were consistent with the predictions of CCM with temporal context represented as 2Tt/(Tt + Ti) and to a lesser extent as (Tt/Ti)k, but not with HVA. Overall, these results suggest that preference increases as a hyperbolic function of Tt/Ti when terminal-link duration is increased relative to initial-link duration, with the terminal-link schedule ratio held constant.

Time and rate measures in choice transitions

Three experiments with pigeons studied the relation between time and rate measures of behavior under conditions of changing preference. Experiment 1 studied a concurrent chain schedule with random-interval initial links and fixed-interval terminal links; Experiment 2 studied a multiple chained random-interval fixed-interval schedule; and Experiment 3 studied simple concurrent random-interval random-interval schedules. In Experiment 1, and to a lesser extent in the other two experiments, session-average initial-link wait-time differences were linearly related to session-average response-rate differences. In Experiment 1, and to a lesser extent in Experiment 3, ratios of session-average initial-link wait times and response rates were related by a power function. The weaker relations between wait and response measures in Experiment 2 appear to be due to the absence of competition between responses. In Experiments 1 and 2, initial-link changes lagged behind terminal-link changes. These findings may have implications for the relations between fixed- and variable-interval procedures and suggest that more attention should be paid to temporal measures in studies of free-operant choice.

Concurrent-chain performance in transition: effects of terminal-link duration and individual reinforcers

Pigeons responded on concurrent-chain schedules with variable-interval initial links and equal delays as terminal links. The terminal-link delays were 1 sec in some conditions and 20 sec in other conditions. The percentages of reinforcers delivered for responses on the left key were 10%, 30%, 70%, or 90%, and this percentage was switched every five to nine sessions. The rate of change in the pigeons' response percentages after a switch was the same whether the terminal-link delays were 1 sec or 20 sec. Analysis of the effects of individual reinforcers showed that after a response on one key had been reinforced, response percentages on that key were higher for at least the next 100 responses. Small effects of individual reinforcers were evident after eight or nine additional reinforcers had been delivered. The effects of individual reinforcers were about equally large during times of transition and during periods in which overall response percentages were relatively stable.

Hyperbolic value addition and general models of animal choice

Three mathematical models of choice--the contextual-choice model (R. Grace, 1994), delay-reduction theory (N. Squires & E. Fantino, 1971), and a new model called the hyperbolic value-added model--were compared in their ability to predict the results from a wide variety of experiments with animal subjects. When supplied with 2 or 3 free parameters, all 3 models made fairly accurate predictions for a large set of experiments that used concurrent-chain procedures. One advantage of the hyperbolic value-added model is that it is derived from a simpler model that makes accurate predictions for many experiments using discrete-trial adjusting-delay procedures. Some results favor the hyperbolic value-added model and delay-reduction theory over the contextual-choice model, but more data are needed from choice situations for which the models make distinctly different predictions.

Choice in the time-left procedure and in concurrent chains with a time-left terminal link

In two experiments, rats chose between a standard fixed-duration food-associated stimulus and a stimulus whose duration was the time remaining to reinforcement in an elapsing comparison interval. In Experiment 1, 4 rats responded in a time-left procedure wherein a single initial-link variable-interval schedule set up two potential terminal links simultaneously. As time elapsed in the initial-link schedule, the choice was between a standard fixed-interval 30-s terminal link and a time-left terminal link whose programmed interval requirement equaled 90 s minus the elapsed time in the initial link. Rats generally responded more on the lever with the shortest programmed terminal-link duration, but the temporal parameters of the procedure were found to vary with response distributions. Contrary to previous reports, therefore, time-left data were well predicted by choice models that make no assumptions about animal timing. In Experiment 2, 8 rats responded on a concurrent-chains schedule with independent variable-interval initial links and a time-left terminal link in one of the choice schedules. On the time-left lever, the programmed terminal-link delay equaled 90 s minus the elapsed time in the time-left initial link. On the standard lever, terminal-link responses were reinforced according to a variable-interval schedule whose average value varied over four conditions. Relative time-left initial-link responses increased in the elapsing time-left initial-link schedule as the time-left terminal link became shorter relative to the standard terminal link. Scalar expectancy theory failed to predict the resultant data, but a modified version of the delay-reduction model made good predictions. An analysis of the elaboration of scalar expectancy theory for variable delays demonstrated that the model is poorly formulated for arithmetically distributed delays.

A quantitative analysis of sensitivity to the conditioned reinforcing value of terminal-link stimuli in a concurrent-chains schedule

Pigeons were exposed to a concurrent-chains schedule in which a single variable-interval 30-s schedule was used in the initial links and fixed-time schedules were used in the terminal links. Three types of keylight conditions were used in the terminal links. In the first condition, different delays were associated with different keylight stimuli (cued condition). In the second condition, different delays were associated with the same stimulus, either a blackout (uncued blackout condition) or a white key (uncued white condition). Paired values of terminal-link fixed-time schedules differed by a constant ratio of 3:1, while the absolute value of delays was varied from 3 s to 54 s. The results showed that choice proportions for the shorter of two delays increased when the absolute size of the delays was increased for all keylight conditions. Further, the choice proportions for the shorter delay increased from the uncued blackout condition, to the uncued white condition, to the cued condition. A modified version of Fantino's (1969) delay-reduction model (expressed as a function relating the response ratio to the delay-reduction ratio) can be applied to these data by showing that sensitivity to delay reduction increased from the uncued blackout condition, to the uncued white condition, to the cued condition. Thus, the present study demonstrated that a modified version of the delay-reduction model can be used to assess quantitative differences in the terminal-link keylight condition in terms of sensitivity to delay reduction (i.e., the conditioned reinforcing value of the terminal-link keylight stimuli).

Choice behavior in transition: development of preference for the higher probability of reinforcement

Ten acquisition curves were obtained from each of 4 pigeons in a two-choice discrete-trial procedure. In each of these 10 conditions, the two response keys initially had equal probabilities of reinforcement, and subjects' choice responses were about equally divided between the two keys. Then the reinforcement probabilities were changed so that one key had a higher probability of reinforcement (the left key in half of the conditions and the right key in the other half), and in nearly every case the subjects developed a preference for this key. The rate of acquisition of preference for this key was faster when the ratio of the two reinforcement probabilities was higher. For instance, acquisition of preference was faster in conditions with reinforcement probabilities of .12 and .02 than in conditions with reinforcement probabilities of .40 and .30, even though the pairs of probabilities differed by .10 in both cases. These results were used to evaluate the predictions of some theories of transitional behavior in choice situations. A trial-by-trial analysis of individual responses and reinforcers suggested that reinforcement had both short-term and long-term effects on choice. The short-term effect was an increased probability of returning to the same key on the one or two trials following a reinforcer. The long-term effect was a gradual increase in the proportion of responses on the key with the higher probability of reinforcement, an increase that usually continued for several hundred trials.