Reinforcement of behavioral patterns: shaping a scallop

The Role of Response-Reward Correlation in Stimulus-Response Overshadowing

Rats pressed levers for food reward which was delivered, when appropriate, 0·4 s after the response. For one group, the delay interval was filled by a light cue; for the other group, the same number of lights was given but they were not correlated with food delivery. In Experiment I, all lever presses were reinforced and there were no differences in response rate between groups. In Experiments II and III, lever pressing was rewarded according to a VI and VR schedule respectively. Group differences were observed in Experiment II but they disappeared in Experiment III. The results of Experiments I and II show that a reward-related stimulus does not overshadow a lever response unless the stimulus is a better predictor of reward. Differences in salience or competition from sign-tracking behaviors were ruled out as causes of this phenomenon. Experiment III demonstrated, however, that a weak response-reward correlation is not a sufficient condition for the overshadowing effect. A fourth experiment replicated the results of Experiment III using naive animals. The results of these last two experiments are not consistent with an information theory approach unless (a) a response-units concept is adopted or (b) the cue involved in overshadowing is not the pre-food light but the end of a temporal interval, whose salience is enhanced by the light.

Toward the Unification of Molecular and Molar Analyses

Three categories of behavior analysis may be called molecular, molar, and unified. Molecular analyses focus on how manual shaping segments moment-to-moment behaving into new, unified, hierarchically organized patterns. Manual shaping is largely atheoretical, qualitative, and practical. Molar analyses aggregate behaviors and then compute a numerical average for the aggregate. Typical molar analyses involve average rate of, or average time allocated to, the aggregated behaviors. Some molar analyses have no known relation to any behavior stream. Molar analyses are usually quantitative and often theoretical. Unified analyses combine automated shaping of moment-to-moment behaving and molar aggregates of the shaped patterns. Unified controlling relations suggest that molar controlling relations like matching confound shaping and strengthening effects of reinforcement. If a molecular analysis is about how reinforcement organizes individual behavior moment by moment, and a molar analysis is about how reinforcement encourages more or less of an activity aggregated over time, then a unified analysis handles both kinds of analyses. Only theories engendered by computer simulation appear to be able to unify all three categories of behavior analysis.

Contingency and behavior analysis

The concept of contingency is central to theoretical discussions of learned behavior and in the application of learning research to problems of social significance. This paper reviews three aspects of the contingency concept as it has been developed by behavior analysts. The first is the empirical analysis of contingency through experimental studies of both human and nonhuman behavior. The second is the synthesis of experimental studies in theoretical and conceptual frameworks to yield a more general account of contingency and to integrate the concept with other behavioral processes. The third aspect is one of practical considerations in the application of the contingency concept in both laboratory and applied settings.

Resurgence of temporal patterns of responding

The resurgence of temporal patterns of key pecking by pigeons was investigated in two experiments. In Experiment 1, positively accelerated and linear patterns of responding were established on one key under a discrete-trial multiple fixed-interval variable-interval schedule. Subsequently, only responses on a second key produced reinforcers according to a variable-interval schedule. When reinforcement on the second key was discontinued, positively accelerated and linear response patterns resurged on the first key, in the presence of the stimuli previously correlated with the fixed- and variable-interval schedules, respectively. In Experiment 2, resurgence was assessed after temporal patterns were directly reinforced. Initially, responding was reinforced if it approximated an algorithm-defined temporal pattern during trials. Subsequently, reinforcement depended on pausing during trials and, when it was discontinued, resurgence of previously reinforced patterns occurred for each pigeon and for 2 of 3 pigeons during a replication. The results of both experiments demonstrate the resurgence of temporally organized responding and replicate and extend previous findings on resurgence of discrete responses and spatial response sequences.

Concurrent performances: rate constancies without changeover delays

Pigeons' pecks on two keys were maintained, without changeover delays, by independent variable-interval schedules of food reinforcement. Four regularly cycling 2-min components scheduled reinforcement respectively for both keys, left key only, both keys, and right key only. Initially, reinforcement scheduled for one key alone produced more responding on that key than reinforcement scheduled concurrently for both keys. Continued sessions reduced this difference; response rate on a given key approached constancy, or invariance with respect to the performance on and schedule for the other key. When extinction replaced the reinforcement schedule on either key, responding on that key decreased more during components that scheduled reinforcement for the other key than during those that did not. This demonstration that responses on one key were not supported by reinforcers on the other key suggested that the alternation of concurrent responding and either-key-alone responding prevented concurrent superstitions from developing.

Cognition, behavior, and the experimental analysis of behavior

A case history illustrates how one research program in the experimental analysis of behavior evolved somewhat differently from the modal research program represented in this journal. A chief issue that seems to be responsible for this difference is the role attributed to theory in behavioral research: Skinner's views on the nature and function of theory and on the nature of observation combine to produce a certain kind of picture of behavior. The classic conception of reinforcement contingencies is tied to this particular picture. But this picture may be incompatible with, and certainly is different from, other possible pictures. Reinforcement contingencies that place greater emphasis on the local temporal patterning of behavior seem tied to some of these alternative pictures of what behavior is. These other pictures encourage a wide range of theoretical approaches, including cognitive ones, various kinds of mathematical analyses, and computer-simulation methods to characterize entire behavior streams. In the future, perhaps the experimental analysis of behavior will accept a somewhat different range of views on the nature and function of theory, a correspondingly different set of experimental methods, and alternative ways of talking about behavior.

Response-reinforcer relations and the maintenance of behavior

The effects on pigeons' key pecking of unsignaled delays of reinforcement and response-independent reinforcement were compared after either variable-interval or differential-reinforcement-of-low-rate baseline schedules. One 30-min session arranging delayed reinforcement and one 30-min session arranging response-independent reinforcement were conducted daily, 6 hr apart. A within-subject yoked-control procedure equated reinforcer frequency and distribution across the two sessions. Response rates usually were reduced more by response-independent than by delayed but response-contingent delivery of reinforcers. Under both schedules, response rates were lower when obtained delays were greater. These results bear upon methodological and conceptual issues regarding comparisons of contingencies that change the temporal response-reinforcer relations.

Rate and temporal pattern of key pecking under autoshaping and omission schedules of reinforcement

The role of response-reinforcer contiguity on autoshaped key pecking in pigeons was studied by scheduling response-dependent nonreinforcement at the beginning or the end of brief (8-sec) discrete trials. Schedules that permitted chance conjunctions of key pecking and food sustained high rates of responding, whereas those that prevented the occurrence of key peck-food intervals shorter than 4 sec sustained low response rates. In addition, selective reinforcement schedules supported accelerating or decelerating rates of responding within individual trials. These effects were traceable to response-reinforcer (operant), but not stimulus-reinforcer (respondent) factors.

Short-term memory in the pigeon: the previously reinforced response

Eighteen pigeons served in a discrete-trials short-term memory experiment in which the reinforcement probability for a peck on one of two keys depended on the response reinforced on the previous trial: either the probability of reinforcement on a trial was 0.8 for the same response reinforced on the previous trial and was 0.2 for the other response (Group A), or, it was 0 or 0.2 for the same response and 1.0 or 0.8 for the other response (Group B). A correction procedure ensured that over all trials reinforcement was distributed equally across the left and right keys. The optimal strategy was either a winstay, lose-shift strategy (Group A) or a win-shift, lose-stay strategy (Group B). The retention interval, that is the intertrial interval, was varied. The average probability of choosing the optimal alternative reinforced 80% of the time was 0.96, 0.84, and 0.74 after delays of 2.5, 4.0, and 6.0 sec, respectively for Group A, and was 0.87, 0.81, and 0.55 after delays of 2.5, 4.0, and 6.0 sec, respectively, for Group B. This outcome is consistent with the view that behavior approximated the optimal response strategy but only to an extent permitted by a subject's short-term memory for the cue correlated with reinforcement, that is, its own most-recently reinforced response. More generally, this result is consistent with "molecular" analyses of operant behavior, but is inconsistent with traditional "molar" analyses holding that fundamental controlling relations may be discovered by routinely averaging over different local reinforcement contingencies. In the present experiment, the molar results were byproducts of local reinforcement contingencies involving an organism's own recent behavior.

Conditioning of two-response patterns of key pecking in pigeons

On discrete trials, two response keys were made available to hungry pigeons and food reinforcement depended on the order in which the required two key pecks occurred. In different phases, only one of the four possible two-peck sequences (left-left, left-right, right-left, and right-right) produced food reinforcement. In each case, the pigeons learned to perform the correct two-peck sequence more often than the incorrect sequences. Furthermore, the course of differentiation mastery indicated that both reinforcement history and response-reinforcer contiguity influenced performance. These results reveal that response patterns comprising two instances of the same response left-left and right-right) or instances of two different responses (left-right and right-left) may function as operants, thereby extending the generality of conditioning principles from discrete responses to structured sequences of behavior. These and other results are discussed in terms of contiguity-based and memory-based models of learning.

Graded differential reinforcement: Response-dependent reinforcer amount

After key pecking had been autoshaped, six pigeons were exposed to a condition in which the duration of grain availability at the end of an 8-second trial depended on the number of responses emitted during the trial (0.25-second access to grain per response). This procedure, called correlated reinforcement, alternated across conditions with the automaintenance baseline in which the 8-second trial terminated with a constant 2.5-second access to grain. Two control procedures were run; in both, the reinforcer durations were yoked to those obtained in the last correlated session. In the yoked control no responses were required, but in the single-response yoked control at least one response was required to receive the yoked duration. The correlated condition maintained response rates above those produced by the two control conditions. These results may be accounted for by differential reinforcement.

Changes in functional response units with briefly delayed reinforcement

In two experiments, key-peck responding of pigeons was compared under variable-interval schedules that arranged immediate reinforcement and ones that arranged unsignaled delays of reinforcement. Responses during the nominal unsignaled delay periods had no effect on the reinforcer presentations. In Experiment 1, the unsignaled delays were studied using variable-interval schedules as baselines. Relative to the immediate reinforcement condition, 0.5-s unsignaled delays decreased the duration of the reinforced interresponse times and increased the overall frequency of short (<0.5-s) interresponse times. Longer, 5.0-s unsignaled delays increased the duration of the reinforced interresponse times and decreased the overall frequency of the short interresponse times. In Experiment 2, similar effects to those of Experiment 1 were obtained when the 0.5-s unsignaled delays were imposed upon a baseline schedule that explicitly arranged reinforcement of short interresponse times and therefore already generated a large number of short interresponse times. The results support earlier suggestions that the unsignaled 0.5-s delays change the functional response unit from a single key peck to a multiple key-peck unit. These findings are discussed in terms of the mechanisms by which contingencies control response structure in the absence of specific structural requirements.

Organization in memory and behavior

SOME COMMON REINFORCEMENT CONTINGENCIES MAKE THE DELIVERY OF A REINFORCER DEPEND ON THE OCCURRENCE OF BEHAVIOR LACKING SIGNIFICANT TEMPORAL STRUCTURE: a reinforcer may be contingent on nearly instantaneous responses such as a pigeon's key peck, a rat's lever press, a human's button press or brief verbal utterance, and so on. Such a reinforcement contingency conforms much more closely to the functionalist tradition in experimental psychology than to the structuralist tradition. Until recently, the functionalist tradition, in the form of a kind of associationism, typified most research on human learning and memory. Recently, however, research on human memory has focused more on structural issues: now the basic unit of analysis often involves an organized temporal pattern of behavior. A focus on the interrelations between the function and structure of behavior identifies a set of independent and dependent variables different from those identified by certain common kinds of "molar" behavioral analyses. In so doing, such a focus redefines some of the significant issues in the experimental analysis of behavior.

Memory for sequences of stimuli and responses

Two experiments sought to determine if pigeons could discriminate and remember recent sequences of stimuli and responses. A variant of Konorski's short-term memory procedure involving successive presentation of sample and test stimuli was used. The samples were stimulus-response pairs of the form, (S-R)(1)-(S-R)(2). Differential test responding disclosed memory of the two-item samples, with birds showing earlier and greater control by the second item than the first (Experiment 1). When the retention interval separating the second item of the sample sequence from the test stimulus was lenghtened from .5 to 2.0 or 4.0 sec, a systematic loss of stimulus control resulted; however, when varied over the same temporal range, the interval between the two items of the sample sequence had a much smaller effect, or none at all (Experiment 2). These results support an account of response-sequence differentiation that stresses short-term memory of organized behavior patterns.

Molecular contingencies: reinforcement probability

Pigeons obtained food by responding in a discrete-trials two-choice probability-learning experiment involving temporal stimuli. A given response alternative, a left- or right-key peck, had 11 associated reinforcement probabilities within each session. Reinforcement probability for a choice was an increasing or a decreasing function of the time interval immediately preceding the choice. The 11 equiprobable temporal stimuli ranged from 1 to 11 sec in 1-sec classes. Preference tended to deviate from probability matching in the direction of maximizing; i.e., the percentage of choices of the preferred response alternative tended to exceed the probability of reinforcement for that alternative. This result was qualitatively consistent with probability-learning experiments using visual stimuli. The result is consistent with a molecular analysis of operant behavior and poses a difficulty for molar theories holding that local variations in reinforcement probability may safely be disregarded in the analysis of behavior maintained by operant paradigms.

Development of complex, stereotyped behavior in pigeons

A pigeon's peck on one key moved a light down one position in a 5x5 matrix of lights, while a peck on another key moved the light across one position. Reinforcement depended upon the occurrence of four pecks on each key (moving the matrix light from the top left to the bottom right), and a fifth peck on either key ended a trial without food. Though there were 70 different sequences that led to reinforcement, each of 12 pigeons developed a particular, stereotyped sequence which dominated its behavior (Experiment 1). Extinction produced substantial increases in sequence variability (Experiment 2). Removal of the matrix cues disrupted performance, though it partially recovered with extended training (Experiment 3). The pigeons did not master a contingency which required a different sequence on the current trial than on the previous one (Experiment 4), though they were able to learn to emit sequences which began with either left-left or left-right response patterns (Experiment 5). The experiments suggest that contingencies of reinforcement may contribute to the creation of complex units of behavior, and that stereotypy may be a likely consequence of contingent reinforcement.

Short-term memory in the pigeon: relative recency

Three pigeons pecked for food in an experiment in which each trial consisted of two phases. The first phase consisted of a pattern of three successively illuminated, randomly selected left or right keys. A subject was required to peck each of the lighted keys as they appeared. Thus, in the first phase, a subject emitted a pattern of three left- or right-key pecks. Over trials, all eight possible patterns appeared. A time interval separated the first phase from the second phase, which began with presentation of a randomly selected one of three cues. A reinforcer was delivered in the second phase if a subject pecked the side key that had appeared in the first phase in an ordinal position corresponding to the cue presented in the second phase. That is, the three cues probed a pigeon's memory for the side key it had pecked first, second, or third, in the first phase of a trial. The results show that a pigeon can remember for more than 4 sec the order in which it has just seen and pecked two lighted keys: a pigeon can remember the temporal organization or pattern of events in its recent environment. Consequently, the functional stimulus present when a reinforcer is delivered may include a subject's short-term memory for the temporal organization of recent events, such as the pattern of its own recent behavior. This possibility is consistent with a molecular analysis of operant behavior focusing on local patterns of behavior.

Memory for recent behavior in the pigeon

Variations of the symbolic delayed-matching-to-sample procedure were used to study a pigeon's memory for a small number of pecks. In the first experiment a choice of a left or right sidekey after a delay or retention interval was reinforced if a bird had not pecked at all or had pecked exactly once, before the delay, respectively. In the second experiment a choice of a red or green sidekey, regardless of its position, was reinforced if a bird had not pecked at all or had pecked exactly twice, respectively. In the first experiment a bird could orient toward the correct choice during the delay, whereas it could not in the second experiment. In a third experiment a feature-probing method was used to study a pigeon's memory for a number of pecks in the context of certain other pecks. The results showed that a pigeon can remember a small number of pecks for one-half to one minute or more and that the percent correct is a decreasing function of the log retention interval. When a second number of pecks is different from the first number, memory for the first number lasts only a few seconds. When a second number is the same, memory lasts considerably longer. The more recent number of pecks is remembered better. The results are interpreted in terms of a theory which holds that a reinforcer, in general, may act on a subjects' memory for recent behavior to generate patterns of behavior.

Some factors that influence the acquisition of complex, stereotyped, response sequences in pigeons

Two pigeons were required to peck six to nine illuminated response keys. A response on any one of the keys darkened that key. When each key had been darkened, a reinforcer was delivered. No specific sequence of key pecking was ever required. The keys were presented in various matrices: three by two, three by three, horizontal rows, and vertical columns. The keys either presented the same stimulus, white light; or each key presented a different stimulus, a color or form. The results indicated that although there were 720 to 362,880 different sequences that would produce reinforcement, each bird developed a particular, stereotyped sequence that dominated its behavior. Variability among the birds across phases yielded less than 60 sequences, .0001 to 6 percent of the possible sequences. The data suggest that a reinforcement contingency that includes "free choice" of response sequence will produce stereotypical response sequences that function as complex "units" of behavior.

On the role of "memory" in the analysis of behavior

Shimp's (1976) recent paper on memory and the structure of behavior is discussed, as is the concept of memory itself. Radical behaviorism is distinguished from associationism, and behavioral accounts of experiments on short-term and long-term memory are provided. Unobservable theoretical quantities, such as rate of response, are distinguished from theoretical constructs such as memory. The logical inconsistency involved in the use of short-term memory as a "theoretical primitive" in the definition of units of behavior is explicated, and an alternative strategy for identifying units of behavior is presented. It is argued that the experimental analysis of behavior has ignored processes usually considered as memorial because many of the phenomena are irreversible. Methods for the study of transitions in single subjects should be developed more fully. Possible pitfalls associated with subscription to the term memory are listed, and it is concluded that the concept is neither required nor helpful in the analysis of behavior.

Conditioning of within-trial patterns of key pecking in pigeons

The possibility of conditioning systematic patterns of responding during brief discrete trials was studied by requiring hungry pigeons to key peck and then pause or to pause and then key peck in order to gain access to food. These schedules were highly effective in promoting decelerated and accelerated rates of responding, respectively, within individual trials; indeed, performance was quite similar to that observed when explicit external stimuli were correlated with "peck" and "pause" portions of the daily trials. Finally, schedules of reinforcement that did not selectively reinforce peck-pause or pause-peck patterns neither generated these patterns nor maintained them at the previous high levels. The results, therefore, confirm Shimp's (1976) proposal that organized groupings of discrete responses may function as operants-even in the absence of strict response-reinforcer contiguity.

Constituents of response rates

Response rate and the proportion of time pigeons allocated to a key-pecking activity were measured on several basic types of reinforcement schedules. Reinforcement frequency was varied within each type of basic schedule, and the effects on two constituents of response rate were noted. Propensity, the proportion of time the birds spent on a platform in front of the key, showed very consistent effects as reinforcement frequency varied: in general, it decreased when reinforcement frequency markedly decreased and it increased when reinforcement frequency increased. Speed, key pecks per unit of time spent on the platform, showed inconsistent effects when reinforcement frequency varied. Consequently, response rate showed less consistent effects than did propensity. Cumulative response records demonstrated the existence of several different types of transitions or boundary states between the key-pecking activity and other activities. The types of transitions that occurred between activities depended on both the type of reinforcement schedule and the frequency of reinforcement. The propensity data support the position that general laws of behavior can be based on temporal measures of behavior. The speed data suggest that, if a complete assessment of the dynamic properties of behavior is to be achieved, measures of behavior must incorporate the structural variations in the operant unit.

Resurgence of integrated behavioral units

Two experiments with rats examined the dynamics of well-learned response sequences when reinforcement contingencies were changed. Both experiments contained four phases, each of which reinforced a 2-response sequence of lever presses until responding was stable. The contingencies then were shifted to a new reinforced sequence until responding was again stable. Extinction-induced resurgence of previously reinforced, and then extinguished, heterogeneous response sequences was observed in all subjects in both experiments. These sequences were demonstrated to be integrated behavioral units, controlled by processes acting at the level of the entire sequence. Response-level processes were also simultaneously operative. Errors in sequence production were strongly influenced by the terminal, not the initial, response in the currently reinforced sequence, but not by the previously reinforced sequence. These studies demonstrate that sequence-level and response-level processes can operate simultaneously in integrated behavioral units. Resurgence and the development of integrated behavioral units may be dissociated; thus the observation of one does not necessarily imply the other.

Molecular order in concurrent response sequences

We studied the order of emission of concurrently reinforced free-operant two-response sequences such as left-left (LL) and left-right (LR). The end of each sequence was demarcated by stimulus change. The use of demarcated sequences of responses, as opposed to individual responses, provides an expanded set of distinct, temporally ordered behaviour pairings to investigate (e.g., LL followed by LL, LL followed by LR, etc.); it is as well a real-life analogue. A sequential analysis of new and existing rat and pigeon data revealed patterns in both overall and post-reinforcer-only sequence emission order. These patterns were consistent across species and individuals, and they followed higher-order organising principles. We describe sequence non-repetition, last-response repetition, and the proportion and post-reinforcer effects, and relate them to existing molar and molecular behaviour principles. Beyond their immediate implications, our results illustrate the value of sequential analysis as a tool for the investigation of molar-molecular behavioural relations.

Preferences among stimulus matches in the pigeon

Three pigeons were trained to peck two to five illuminated response keys. A peck to any of the keys changed the stimulus on the key. When all keys showed the same stimulus (i.e., a stimulus match), an additional key was illuminated with white light. A peck on this key produced three-second access to grain, a three-second intertrial interval, and the next trial. For most sessions, no particular stimulus match was required. Although there were often several stimuli available, each bird preferred a particular stimulus match. With up to 12 stimuli available, birds matched a particular stimulus 60% to 100% of the time.

Matching, maximizing, and the behavioral unit: concurrent reinforcement of response sequences

Pigeons pecked two keys in a probability matching situation in which four two-peck sequences were intermittently reinforced: left-left, left-right, right-left and right-right. In Phase 1, relative reinforcement rate was varied with respect to the first response of a sequence: reinforcers were differentially assigned for left-left and left-right sequences as opposed to right-left and right-right sequences. The second response of reinforced sequences occurred equally on the left and right keys across conditions. In Phase II, relative reinforcement rate was varied for sequences that involve an alternation as opposed to those that did not. The relative outputs of the different sequences matched the relative reinforcement rates for the different sequences in both phases. Relative response rates for key pecks did not always match relative reinforcement rates. The intertrial interval separating responses was varied in both phases; increases in the intertrial interval affected the relative frequency of different sequences. The results demonstrate that response sequences acted as functional units influencing choice and thus support a structural account of choice. At the same time, the matching of relative sequence proportion and relative reinforcement rate supports a matching account.

Short-term memory in the pigeon: delayed-pair-comparison procedures and some results

A discrete-trials, delayed-pair-comparison procedure was developed to study visual short-term memory for tilted lines. In four experiments, pigeons' responses on left or right keys were reinforced with food depending on whether a comparison stimulus was or was not the same as a standard stimulus presented earlier in the same trial. In Experimental I, recall was an increasing function of the exposure time of the to-be-remembered stimulus and was a decreasing function of the retention interval. In Experiment II, retroactive interference was investigated: recall was poorer after a retention interval during which was presented either a tilted line or contextual stimuli in the form of the illuminated experimental chamber. In Experiment III, a subject was required to engage, throughout the retention interval, in one or the other of two different behaviors, depending on which of two stimuli a subject was to remember. This mnemonic strategy vastly improved recall after 15- and 20-second retention intervals. In Experiment IV, the opposite end of the performance continuum was studied: by combining the effects of a larger stimulus set and the effects of what presumably was an increased memory load, performance was reduced to approximately chance levels after retention intervals shorter than 1 second.