Matching, maximizing, and the behavioral unit: concurrent reinforcement of response sequences

A Critical Review of the Support for Variability as an Operant Dimension

There is abundant evidence that behavioral variability is more predominant when reinforcement is contingent on it than when it is not, and the interpretation of direct reinforcement of variability suggested by Page and Neuringer, Journal of Experimental Psychology: Animal Behavior Processes, 11(3), 429-452 (1985) has been widely accepted. Even so, trying to identify the underlying mechanisms in the emergence of stochastic-like variability in a variability contingency is intricate. There are several challenges to characterizing variability as directly reinforced, most notably because reinforcement traditionally has been found to produce repetitive responding, but also because directly reinforced variability does not always relate to independent variables the same way as more commonly studied repetitive responding does. The challenging findings in variability experiments are discussed, along with alternative hypotheses on how variability contingencies may engender the high variability that they undeniably do. We suggest that the typical increase in behavioral variability that is often demonstrated when reinforcement is contingent on it may be better explained in terms of a dynamic interaction of reinforcement and extinction working on several specific responses rather than as directly reinforced.

Choosing to Vary and Repeat

Can animals choose to vary their behaviors or to repeat them, depending on the rewards they earn for behaving variably versus repetitively? To answer this question, pigeons were rewarded for four-response sequences made to left (L) and right (R) disks. A “varied” sequence differed from each of the previous three sequences, and a “repeated” sequence was the same as some one of the previous three. For example, if a pigeon had generated sequences LLLL. LLLR, and LLRR in that order, then an RRRR sequence in the next trial was defined as a variation, whereas LLLL was a repetition. Two experiments showed that frequencies of varied and repeated sequences depended on the frequencies with which they were reinforced, with a “matching” relationship accounting for the results. It was concluded that pigeons' choices to vary or repeat parallel their choices between simpler response alternatives, a result consistent with the hypothesis that behavioral variability is influenced by its consequences. This finding may help to explain the “voluntary” or “free” nature of operant behavior.

Learning new response sequences

Four rats were required to press either a right or left lever to complete various three-response sequences. After extended exposure to a training sequence, subjects were shifted to a new target sequence. The new target sequences always differed from the previous sequence by the response required in the first or last position of the sequence. Subjects were repeatedly exposed to all possible combinations of training and new target sequences. Learning of new sequences occurred more rapidly when the change in the new target sequence was in the last position. Errors persisted longer in new sequences in which the change was in the first position. Extinction of the training sequence occured faster when the change was in the last position. Responses in the last position were considerably more sensitive to the shift to new target sequences than were responses in the first position. Even though response sequences may form new behavioral units from the training sequence, reinforcement and extinction acted differentially on the individual lever presses within new target sequences rather than on the sequences as a whole. These findings support the hypothesis that response strength is determined by contiguity to reinforcement.

Choice and behavioral patterning

Ten pigeons pecked left and right keys in a discrete-trials experiment in which access to food was contingent upon changeovers to the right key after particular runs of left-key pecks. In each of three sets of conditions, two run lengths were reinforced according to a concurrent variable-interval schedule: reinforcement followed runs of either 1 or 2, 1 or 4, or 2 or 4 left-key pecks preceding changeovers. The intertrial interval separating successive pecks was varied from .5 to 10.0 sec, and the relative frequency of reinforcement for the shorter of the two reinforced runs was varied from 0 to .75. The contingencies established local behavioral patterning that roughly approximated that required for reinforcement. For a fixed pair of reinforced run lengths, preference for the shorter of the two frequently increased as the intertrial interval increased and therefore as the minimum temporal durations of both reinforced runs increased. Preference for the shorter of the two also increased as its corresponding relative frequency of reinforcement increased. Both of these effects on preference were qualitatively similar to corresponding effects in previous research with two different kinds of reinforced behavioral patterns, interresponse times and interchangeover times. In all these experiments, analytical units were found in the temporal patterns of behavior, not in the behavior immediately contiguous with a reinforcer. It is suggested that a particular local temporal pattern of behavior is established to the extent to which it is repeatedly remembered when reinforcers are delivered, regardless of whether the delivery of a reinforcer is explicitly contingent upon that pattern.

Sequences of spaced responses: Behavioral units and the role of contiguity

Sequences of temporally spaced responses were reinforced to investigate the effects of delay of reinforcement on the formation of functional behavioral units. In Experiment 1, rats' two- and three-response demarcated sequences of left and right lever presses were reinforced such that different response distributions would occur depending on whether the sequences themselves or individual responses were functional units. The matching law could thus be obeyed either by individual responses or by sequences, but not by both; intermediate results were possible. Both regular (nonretractable) and retractable levers were used; the retractable levers precluded the occurrence of insufficiently spaced responses. At a minimum interresponse time of 5 s for regular levers and 7 s for retractable ones, matching results were intermediate, with greater evidence of sequence conditionability in the two-response sequences than in the three-response sequences. In Experiment 2, the required minimum interresponse spacing for two-response retractable-lever sequences was varied in an attempt to locate the sequence matching threshold. This attempt was unsuccessful, but the sequences (instead of individual responses) more closely obeyed the matching law. In the shortest spaced condition, conditional probability data on Lag 1 sequence emission order showed marked, highly similar patterning for all rats, indicating sequential control of the sequences. Post hoc definition of the behavioral unit in these studies is ambiguous. Although reinforcement contiguity was important, aspects of the results could support both molar- and molecular-level interpretations.

Allocation of complex, sequential operants on multiple and concurrent schedules of reinforcement

Pigeons could produce food by pecking exactly four times on each of two keys, in any order. In the first experiment, these response sequences were reinforced on a series of multiple schedules of variable-interval reinforcement. In the second experiment, these response sequences were reinforced on a series of concurrent schedules of reinforcement. In both experiments, highly stereotyped response sequences developed. If these response sequences were treated as individual responses, the resulting data conformed to what is typically reported in studies of multiple and concurrent schedules involving individual responses. For example, behavioral contrast was observed with the multiple schedules, and matching was observed with the concurrent schedules. However, schedule manipulation had no effect on within-sequence characteristics of responses like accuracy, stereotypy, or rate. These data constitute further evidence that response sequences can become functional behavioral units.

Choice with a fixed requirement for food, and the generality of the matching relation

Pigeons were trained on choice procedures in which responses on each of two keys were reinforced probabilistically, but only after a schedule requirement had been met. Under one arrangement, a fixed-interval choice procedure was used in which responses were not reinforced until the interval was over; then a response on one key would be reinforced, with the effective key changing irregularly from interval to interval. Under a second, fixed-ratio choice procedure, responses on either key counted towards completion of the ratio and then, once the ratio had been completed, a response on the probabilistically selected key would produce food. In one experiment, the schedule requirements were varied for both fixed-interval and fixed-ratio schedules. In the second experiment, relative reinforcement rate was varied. And in a third experiment, the duration of an intertrial interval separating choices was varied. The results for 11 pigeons across all three experiments indicate that there were often large deviations between relative response rates and relative reinforcement rates. Overall performance measures were characterized by a great deal of variability across conditions. More detailed measures of choice across the schedule requirement were also quite variable across conditions. In spite of this variability, performance was consistent across conditions in its efficiency of producing food. The absence of matching of behavior allocation to reinforcement rate indicates an important difference between the present procedures and other choice procedures; that difference raises questions about the specific conditions that lead to matching as an outcome.

Levels of aggregation: Relative time allocation in concurrent-schedule performance

The proportion of time allocated to one component of a concurrent variable-interval variable-interval schedule was computed for groups of interchangeover times (aggregates) within several intact time series. Variability in obtained proportions decreased as the number of interchangeover times within each aggregate increased; however, modal proportions failed to correspond to overall relative time allocation computed over the course of an entire experimental session, even at the largest aggregate size. The aggregated time series showed periodicities at small aggregate sizes and general trends in local preference at larger aggregate sizes. It is suggested that overall relative time allocation represents a molar extreme in the aggregation of behavior that may not accurately reflect central tendency in the allocation of time to available alternatives within the context of ongoing behavior.

Resurgence of behavior during extinction depends on previous rate of response

In two experiments, food-deprived rat subjects leverpressed for food in three successive training phases. In the first phase of both experiments, rats were exposed to a multiple schedule, one component of which produced a high rate of response, and the other of which produced a lower rate of response (multiple random ratio [RR], random interval [RI] in Experiment 1, and multiple differential reinforcement of high rate, differential reinforcement of low rate in Experiment 2). Rats were then transferred to a multiple fixed interval (FI; 60-sec, 60-sec) schedule, until the effects of the first phase on response rate were no longer apparent and their response rates did not differ from those of rats responding on a multiple FI 60-sec, FI 60-sec schedule without previously experiencing a multiple RR, RI schedule. During the third stage oftraining, all rats were placed into extinction. During extinction, rates of responding were higher in the component previously associated with the high rate of responding in Phase 1, and they were lower in the component previously associated with low rates of responding in Phase 1. These results suggest that resurgence effects, like other history effects, are controlled by previous rates of responding.

Resurgence of integrated behavioral units

Two experiments with rats examined the dynamics of well-learned response sequences when reinforcement contingencies were changed. Both experiments contained four phases, each of which reinforced a 2-response sequence of lever presses until responding was stable. The contingencies then were shifted to a new reinforced sequence until responding was again stable. Extinction-induced resurgence of previously reinforced, and then extinguished, heterogeneous response sequences was observed in all subjects in both experiments. These sequences were demonstrated to be integrated behavioral units, controlled by processes acting at the level of the entire sequence. Response-level processes were also simultaneously operative. Errors in sequence production were strongly influenced by the terminal, not the initial, response in the currently reinforced sequence, but not by the previously reinforced sequence. These studies demonstrate that sequence-level and response-level processes can operate simultaneously in integrated behavioral units. Resurgence and the development of integrated behavioral units may be dissociated; thus the observation of one does not necessarily imply the other.

Resurgence of response sequences during extinction in rats shows a primacy effect

Rats were trained to emit a series of three-response sequences to a criterion (i.e., more than 80% of all emitted sequences correct over five successive sessions). Each rat was trained on a series of different, three-response sequences. After the final three-response sequence was acquired, two extinction tests were administered, and the three-response sequences that re-emerged during these extinction tests were noted. Resurgence effects during extinction were observed; that is, the previously trained sequences were emitted. These resurgence effects followed an orderly pattern, which involved a primacy effect. The rats initially emitted the immediately previously trained response, but then started to emit the response sequence they first were trained to emit. Thus, resurgence behavior during extinction can be an orderly function of previous training history. These results replicate those previously obtained with human subjects.

The development of functional response units: the role of demarcating stimuli

An experiment with rats examined the roles of demarcating stimuli and differential reinforcement probability on the development of functional response units. It examined the development of units in a probabilistic, free-operant situation in which the presence of demarcating stimuli was manipulated. In all conditions, behavior became organized into two-response sequences framed by changes in local reinforcement probability. A tone demarcating the beginning and end of contingent response sequences facilitated the development of functional response units, as in chunking, but the same units developed slowly in the absence of the tone. Complex functional response units developed even though reinforcement contigencies remained constant. These findings demonstrate that models of operant learning must include a mechanism for changing the response unit as a function of reinforcement history. Markov models may seem to be a natural technique for modeling response sequences because of their ability to predict individual responses as a function of reinforcement history; however, no class of Markov chain can incorporate changing response units in their predictions.

Duration comparison: relative stimulus differences stimulus age, and stimulus predictiveness

Under a psychophysical trials procedure, pigeons were presented with a red light of one duration followed by a green light of a second duration. Eight geometrically spaced base durations were paired with one of four shorter and four longer durations as the alternate member of a duration pair, with different pairs randomly intermixed. One choice was reinforced if red had lasted longer than green, and a second choice was reinforced if green had lasted longer. Performance was compared when all the base durations and their pair members were included (entire-range condition) or when only the four longest base durations and their comparison durations (restricted-range condition) were used. Discrimination sensitivity decreased for longer duration pairs under both conditions, supporting a memory-based account. Sensitivity was lower under the restricted-range condition. Under both conditions, a bias to report "green as longer" increased as the second green duration increased. Bias changed as a matching function of the green-duration predictiveness of the correct choice. The results are related to a quantitative model of timing and remembering proposed by Staddon.

Behavioral variability and frequency-dependent selection

In Experiment 1, two conditions were compared: (a) a variability schedule in which food reinforcement was delivered for the fourth peck in a sequence that differed from the preceding N four-peck sequences, with the value of N continuously adjusted to maintain reinforcement probability approximately constant; and (b) a control condition in which the variability constraint was dropped but reinforcement probability remained constant. Pigeons responded approximately randomly under the variability schedule but showed strong stereotyped behavior under the control condition. Experiments 2 and 3 tested the idea that variability is the outcome of a type of frequency-dependent selection, namely differential reinforcement of infrequent behavior patterns. The results showed that pigeons alternate when frequency-dependent selection is applied to single pecks because alternation is an easy-to-learn stable pattern that satisfies the frequency-dependent condition. Nevertheless, 2 of 4 pigeons showed random behavior when frequency-dependent selection was applied to two pecks, even though double alternation is a permissible and stable stereotype under these conditions. It appears that random behavior results when pigeons are unable to acquire the stable stereotyped behavior under a given frequency-dependent schedule.