Concurrent schedules: Discriminating reinforcer-ratio reversals at a fixed time after the previous reinforcer

Choosing a future from a murky past: A generalization-based model of behavior

Remembering the past appears critical in allowing organisms to detect order in an environment, and hence to behave in accordance with likely future events. Yet the shortcomings of remembering and perceiving typically mean that the remembered past differs from the actual past, and hence that behavior does not perfectly track the structure of the environment. Here, we outline how the process of generalization might be used to understand differences between what an organism does, and the structure of the past and potential structure of the environment. We explore how different sources of generalization - both from within the same stimulus situation, and from different stimulus situations - might be modeled quantitatively, and how predictions made by this modeling approach are supported by research. Finally, we discuss how generalization from multiple stimulus situations, longer-term experience, and from stimulus situations in the past that are not identical to the stimulus situation in the present, might contribute to our understanding of how an organism's experience translates into behavior.

Modeling choice across time: Effects of response-reinforcer discriminability

This experiment asks whether timing is affected by animals' discrimination of response-reinforcer contingencies, and if so, how this effect can be understood. Six pigeons were trained on a procedure in which concurrent-schedule reinforcer ratios between left and right keys changed at 30 s after the last reinforcer. One stimulus signaled a reinforcer-ratio reversal from 9:1 to 1:9 on that key, and the other stimulus signaled the inverse reversal, with the key on which these stimuli occurred randomized. Across conditions, the physical difference between the stimuli signaling the two responses was varied and the directional changes in the reinforcer ratio signaled by each stimulus were reversed. Choice changed appropriately across time when the two stimuli were discriminable, and points of subjective equality fell with decreasing stimulus difference. A model which assumed that reinforcers obtained in time bins were redistributed across other time bins according to ogivally changing standard deviations, and between response locations according to an ogivally changing redistribution measure, accounted well for the data. This model was shown to be preferable to one in which across-time redistributions were scalar, and across-location redistribution was constant. These results show the critical importance of stimulus-response-reinforcer discriminability to measures of timing.

Generalizing from the Past, Choosing the Future

Behavior in the present depends critically on experience in similar environments in the past. Such past experience may be important in controlling behavior not because it determines the strength of a behavior, but because it allows the structure of the current environment to be detected and used. We explore a prospective-control approach to understanding simple behavior. Under this approach, order in the environment allows even simple organisms to use their personal past to respond according to the likely future. The predicted future controls behavior, and past experience forms the building blocks of the predicted future. We explore how generalization affects the use of past experience to predict and respond to the future. First, we consider how generalization across various dimensions of an event determines the degree to which the structure of the environment exerts control over behavior. Next, we explore generalization from the past to the present as the method of deciding when, where, and what to do. This prospective-control approach is measurable and testable; it builds predictions from events that have already occurred, and assumes no agency. Under this prospective-control approach, generalization is fundamental to understanding both adaptive and maladaptive behavior.

Being there on time: Reinforcer effects on timing and locating

In research on timing, reinforcers often are assumed to influence discrimination of elapsed time. We asked whether changes in choice used to measure timing arise because of joint control by elapsed time and reinforcers, rather than from the direct modification of control by elapsed time by reinforcers. Pigeons worked on a concurrent-choice task in which 1 response was 9 times more likely to produce a reinforcer, reversing between locations when 19 s had elapsed since the marker event. Across conditions, we manipulated the percentage of reinforcers arranged before the probability reversal from 5 to 95%. These changes in reinforcer percentages altered control by location-based elements of the contingency, but not by time-based elements. Choice was well described by a model that assumes that control by the contingency is weakened by generalization across the time and location of reinforcers, and that these generalizations become more likely at later times since a marker. These findings add to a growing body of research that suggests that reinforcers share the same function as other environmental events in determining how the environment controls behavior.

Environment tracking and signal following in a reinforcer-ratio reversal procedure

Several studies suggest that the degree of control by reinforcer ratios (environment tracking) and by exteroceptive stimuli that signal future reinforcer availability (signal following) depends on environmental certainty: As reinforcers become more likely at one location, environmental contingencies exert stronger control and exteroceptive stimuli exert weaker control. This research has not yet been extended to environments in which reinforcer availability changes across time, even though such changes are present in most natural environments. Thus, in the present experiment, we examined environment tracking and signal following in a concurrent schedule in which the reinforcer ratio reversed to its reciprocal 30 s after a reinforcer delivery and keylight-color stimuli signaled the likely or definite time or location of the next reinforcer. Across conditions, we manipulated environmental certainty by varying the probability of reinforcer deliveries on the locally richer key. This made the location of future reinforcers at a particular time more or less certain, but did not change the overall reinforcer ratio. Changes in local environmental certainty had little to no effect on environment tracking and signal following; in all conditions, keylight-color stimuli strongly controlled choice and reinforcer ratios exerted weak control. The present findings suggest that the extent of environment tracking and signal following is primarily determined by global, not local, environmental certainty.

Quantitative analysis of local-level resurgence

Resurgence is the recurrence of a previously reinforced and then extinguished behavior induced by the extinction of another more recently reinforced behavior. Resurgence provides insight into behavioral processes relevant to treatment relapse of a range of problem behaviors. Resurgence is typically studied across three phases: (1) reinforcement of a target response, (2) extinction of the target and concurrent reinforcement of an alternative response, and (3) extinction of the alternative response, resulting in the recurrence of target responding. Because each phase typically occurs successively and spans multiple sessions, extended time frames separate the training and resurgence of target responding. This study assessed resurgence more dynamically and throughout ongoing training in 6 pigeons. Baseline entailed 50-s trials of a free-operant psychophysical procedure, resembling Phases 1 and 2 of typical resurgence procedures. During the first 25 s, we reinforced target (left-key) responding but not alternative (right-key) responding; contingencies reversed during the second 25 s. Target and alternative responding followed the baseline reinforcement contingencies, with alternative responding replacing target responding across the 50 s. We observed resurgence of target responding during signaled and unsignaled probes that extended trial durations an additional 100 s in extinction. Furthermore, resurgence was greater and/or sooner when probes were signaled, suggesting an important role of discriminating transitions to extinction in resurgence. The data were well described by an extension of a stimulus-control model of discrimination that assumes resurgence is the result of generalization of obtained reinforcers across space and time. Therefore, the present findings introduce novel methods and quantitative analyses for assessing behavioral processes underlying resurgence.

The non-Darwinian evolution of behavers and behaviors

Many readers of this journal have been schooled in both Darwinian evolution and Skinnerian psychology, which have in common the vision of powerful control of their subjects by their sequalae. Individuals of species that generate more successful offspring come to dominate their habitat; responses of those individuals that generate more reinforcers come to dominate the repertoire of the individual in that context. This is unarguable. What is questionable is how large a role these forces of selection play in the larger landscape of existing organisms and the repertoires of their individuals. Here it is argued that non-Darwinian and non-Skinnerian selection play much larger roles in both than the reader may appreciate. The argument is based on the history of, and recent advances in, microbiology. Lessons from that history re-illuminate the three putative domains of selection by consequences: The evolution of species, response repertoires, and cultures. It is argued that before, beneath, and after the cosmically brief but crucial epoch of Darwinian evolution that shaped creatures such as ourselves, non-Darwinian forces pervade all three domains.

A model for discriminating reinforcers in time and space

Both the response-reinforcer and stimulus-reinforcer relation are important in discrimination learning; differential responding requires a minimum of two discriminably-different stimuli and two discriminably-different associated contingencies of reinforcement. When elapsed time is a discriminative stimulus for the likely availability of a reinforcer, choice over time may be modeled by an extension of the Davison and Nevin (1999) model that assumes that local choice strictly matches the effective local reinforcer ratio. The effective local reinforcer ratio may differ from the obtained local reinforcer ratio for two reasons: Because the animal inaccurately estimates times associated with obtained reinforcers, and thus incorrectly discriminates the stimulus-reinforcer relation across time; and because of error in discriminating the response-reinforcer relation. In choice-based timing tasks, the two responses are usually highly discriminable, and so the larger contributor to differences between the effective and obtained reinforcer ratio is error in discriminating the stimulus-reinforcer relation. Such error may be modeled either by redistributing the numbers of reinforcers obtained at each time across surrounding times, or by redistributing the ratio of reinforcers obtained at each time in the same way. We assessed the extent to which these two approaches to modeling discrimination of the stimulus-reinforcer relation could account for choice in a range of temporal-discrimination procedures. The version of the model that redistributed numbers of reinforcers accounted for more variance in the data. Further, this version provides an explanation for shifts in the point of subjective equality that occur as a result of changes in the local reinforcer rate. The inclusion of a parameter reflecting error in discriminating the response-reinforcer relation enhanced the ability of each version of the model to describe data. The ability of this class of model to account for a range of data suggests that timing, like other conditional discriminations, is choice under the joint discriminative control of elapsed time and differential reinforcement. Understanding the role of differential reinforcement is therefore critical to understanding control by elapsed time.