Intertrial interval as a contextual stimulus: further analysis of a novel asymmetry in temporal discrimination learning

Theory of reinforcement schedules

The three principles of reinforcement are (1) events such as incentives and reinforcers increase the activity of an organism; (2) that activity is bounded by competition from other responses; and (3) animals approach incentives and their signs, guided by their temporal and physical conditions, together called the "contingencies of reinforcement." Mathematical models of each of these principles comprised mathematical principles of reinforcement (MPR; Killeen, 1994). Over the ensuing decades, MPR was extended to new experimental contexts. This article reviews the basic theory and its extensions to satiation, warm-up, extinction, sign tracking, pausing, and sequential control in progressive-ratio and multiple schedules. In the latter cases, a single equation balancing target and competing responses governs behavioral contrast and behavioral momentum. Momentum is intrinsic in the fundamental equations, as behavior unspools more slowly from highly aroused responses conditioned by higher rates of incitement than it does from responses from leaner contexts. Habits are responses that have accrued substantial behavioral momentum. Operant responses, being predictors of reinforcement, are approached by making them: The sight and feel of a paw on a lever is approached by placing paw on lever, as attempted for any sign of reinforcement. Behavior in concurrent schedules is governed by approach to momentarily richer patches (melioration). Applications of MPR in behavioral pharmacology and delay discounting are noted.

BEHAVIORAL AND NEUROBIOLOGICAL MECHANISMS OF PAVLOVIAN AND INSTRUMENTAL EXTINCTION LEARNING

This article reviews the behavioral neuroscience of extinction, the phenomenon in which a behavior that has been acquired through Pavlovian or instrumental (operant) learning decreases in strength when the outcome that reinforced it is removed. Behavioral research indicates that neither Pavlovian nor operant extinction depends substantially on erasure of the original learning but instead depends on new inhibitory learning that is primarily expressed in the context in which it is learned, as exemplified by the renewal effect. Although the nature of the inhibition may differ in Pavlovian and operant extinction, in either case the decline in responding may depend on both generalization decrement and the correction of prediction error. At the neural level, Pavlovian extinction requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in extinction retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. Instrumental extinction involves distinct ensembles in corticostriatal, striatopallidal, and striatohypothalamic circuits as well as their thalamic returns for inhibitory (extinction) and excitatory (renewal and other relapse phenomena) control over operant responding. The field has made significant progress in recent decades, although a fully integrated biobehavioral understanding still awaits.

Retrieval practice after multiple context changes, but not long retention intervals, reduces the impact of a final context change on instrumental behavior

Recent evidence from this laboratory suggests that a context switch after operant learning consistently results in a decrement in responding. One way to reduce this decrement is to train the response in multiple contexts. One interpretation of this result, rooted in stimulus sampling theory, is that conditioning of a greater number of common stimulus elements arising from more contexts causes better generalization to new contexts. An alternative explanation is that each change of context causes more effortful retrieval, and practice involving effortful retrieval results in learning that is better able to transfer to new situations. The current experiments were designed to differentiate between these two explanations for the first time in an animal learning and memory task. Experiment 1 demonstrated that the detrimental impact of a context change on an instrumental nose-poking response can be reduced by training the response in multiple contexts. Experiment 2 then found that a training procedure which inserted extended retention intervals between successive training sessions did not reduce the detrimental impact of a final context change. This occurred even though the inserted retention intervals had a detrimental impact on responding (and, thus, presumably retrieval) similar to the effect that context switches had in Experiment 1. Together, the results suggest that effortful retrieval practice may not be sufficient to reduce the negative impact of a context change on instrumental behavior. A common elements explanation which supposes that physical and temporal contextual cues do not overlap may account for the findings more readily.

Extinction of instrumental (operant) learning: interference, varieties of context, and mechanisms of contextual control

This article reviews recent research on the extinction of instrumental (or operant) conditioning from the perspective that it is an example of a general retroactive interference process. Previous discussions of interference have focused primarily on findings from Pavlovian conditioning. The present review shows that extinction in instrumental learning has much in common with other examples of retroactive interference in instrumental learning (e.g., omission learning, punishment, second-outcome learning, discrimination reversal learning, and differential reinforcement of alternative behavior). In each, the original learning can be largely retained after conflicting information is learned, and behavior is cued or controlled by the current context. The review also suggests that a variety of stimuli can play the role of context, including room and apparatus cues, temporal cues, drug state, deprivation state, stress state, and recent reinforcers, discrete cues, or behaviors. In instrumental learning situations, the context can control behavior through its direct association with the reinforcer or punisher, through its hierarchical relation with response-outcome associations, or its direct association (inhibitory or excitatory) with the response. In simple instrumental extinction and habit learning, the latter mechanism may play an especially important role.

Superior ambiguous occasion setting with visual than temporal feature stimuli

Three experiments with rats compared the relative ease with which different sets of visual or temporal cues could participate in Pavlovian learning. In Experiment 1, 1 group was trained to discriminate between visual cues (Light vs. Dark), whereas the other group learned to discriminate between temporal cues (early [10 s] vs. late [90 s]). Both groups learned to distinguish food-paired from nonpaired periods equally well. In Experiment 2, 2 groups were trained on an ambiguous occasion setting task. For Group Visual, a 2-min Light period signaled that 1 10-s auditory conditioned stimulus, CS1, was reinforced with 1 unconditioned stimulus, US1, but that CS2 was not reinforced; whereas a 2-min dark period signaled that CS1 was not reinforced, but CS2 was reinforced with US2 (i.e., Light: CS1-US1, CS2-; Dark: CS1-, CS2-US2). For Group Temporal, early (10-s) or late (90-s) temporal cues within each of these Light and Dark periods were diagnostic of these contingencies (i.e., Early: CS1-US1, CS2-; Late: CS1-, CS2-US2). Group Visual learned the task, but Group Temporal did not. In Experiment 3 we demonstrated that animals could not solve a related temporal ambiguous occasion setting task in which 1 visual stimulus signaled that both CSs were reinforced early whereas the other visual stimulus signaled that the CSs were reinforced only late. Contrary to a currently popular information theory approach to timing in Pavlovian learning, these results suggest that overt nontemporal visual stimuli are better incorporated into conditional discrimination learning than are temporal stimuli. (PsycINFO Database Record

Temporal generalization gradients following an interdimensional discrimination protocol

We investigated the effects of interdimensional discrimination training in the temporal generalization gradient. In a matching-to-sample task, pigeons learned to choose key S after a T-s houselight sample and key NS in the absence of the houselight sample. For one group of pigeons, T = 20 s; for another, T = 10 s. Subsequently, houselight duration was varied to obtain temporal generalization gradients. Results showed that (a) proportion S increased as houselight duration ranged from 0 s to T s and then remained high for houselight durations longer than T; (b) the gradients were well described by negative-exponential functions; (c) these non-flat gradients were present from the beginning of testing, and; (d) the average gradients obtained with T = 20 s and T = 10 s overlapped when plotted in relative time. We conclude that temporal control does not require explicit discrimination training along the temporal dimension, and that temporal generalization gradients obtained with an interdimensional protocol show the scalar property of timing. We discuss how these findings challenge current models of timing.

Asymmetry in the discrimination of quantity: The role of stimulus generalization

In order to evaluate 1 account for the asymmetry that has been found with discriminations based on stimulus magnitude, in 5 autoshaping experiments, 2 groups of pigeons received a discrimination between 5 and 20 squares presented on a TV screen. One group received a 20+/5- discrimination, with food signaled by 20 squares but not 5 squares; the other group received the opposite discrimination, 5+/20-. The 20+/5- discrimination was acquired more readily than 5+/20- in Experiments 1, 3a, 3b, and 4. For Experiment 1, the screen was white for the intertrial interval (ITI) and the stimuli were black squares on a white background; for Experiment 3a, the screen was black for the ITI and the stimuli were black squares on a white background; and for Experiments 3b and 4, the screen was white for the ITI and the stimuli were white squares on a black background. In Experiment 2, the stimuli were black squares on a white background, but they were separated by an ITI in which 288 black squares were presented against a white background. The 20+/5- discrimination was now acquired more slowly than the 5+/20- discrimination. The asymmetry in the acquisition of the magnitude discriminations in each experiment is attributed to inhibition being associated with the stimuli present during the ITI. The generalization of this inhibition, along a dimension related to the number of squares on the screen, is then assumed to disrupt the acquisition of 1 discrimination to a greater extent than the other.

Asymmetry in the discrimination of quantity by rats: The role of the intertrial interval

In three experiments, rats were trained to discriminate between 20 and five (Exps. 1 and 2), or between 40 and five (Exp. 3), black squares. The squares were randomly distributed in the center of a white background and displayed on a computer screen. For one group, the patterns containing the higher quantity of squares signaled the delivery of sucrose (+), whilst patterns with the lower quantity of squares did not (-). For the second group, sucrose was signaled by the lower, but not by the higher, quantity of squares. In Experiment 1, the intertrial interval (ITI) was a white screen, and the 20+/5- discrimination was acquired more readily than the 5+/20- discrimination. For Experiment 2, the ITI was made up of 80 black squares on a white background. In this instance, the 5+/20- discrimination was acquired more successfully than the 20+/5- discrimination. In Experiment 3, two groups were trained with a 40+/5- discrimination, and two with a 5+/40- discrimination. For one group from each of these pairs, the training trials were separated by a white ITI, and the 40+/5- discrimination was acquired more readily than the 5+/40- discrimination. For the remaining two groups, the training trials were not separated by an ITI, and the two groups acquired the task at approximately the same rate. The results indicate that the cues present during the ITI play a role in the asymmetrical acquisition of magnitude discriminations based on quantity.

Asymmetrical generalization of length in the rat

Two groups of rats in Experiment 1 were required to escape from a square pool by swimming to 1 of 2 submerged platforms that were situated beside the centers of 2 opposite walls. To help rats find a platform, black panels of equal width were pasted to the middle of the walls that were adjacent to the platforms. The width of the 2 panels was 50 cm for Group 50, and 100 cm for Group 100. Test trials were then conducted in the same pool, but with the platforms removed and with a 50-cm panel on 1 wall and a 100-cm panel on the opposite wall. Group 50 expressed a stronger preference for the 100-cm than the 50-cm panel during the test, whereas Group 100 expressed a similar preference for both panels. Thus the degree of generalization from the short to the long panel was greater than from the long to the short panel. Experiments 2 and 3 pointed to the same conclusion. They were of a similar design to Experiment 1, except that the lengths of the panels for the 2 groups were 25 and 50 cm in Experiment 2, and 25 and 100 cm in Experiment 3. The results are explained by assuming the original training results in the walls without black panels entering into inhibitory associations. This inhibition is then assumed to generalize more to the short than the long test panels and thereby result in an asymmetry in the gradients of generalization between the different lengths.

Temporal Averaging Across Stimuli Signaling the Same or Different Reinforcing Outcomes in the Peak Procedure

The present study examined factors that affect temporal averaging in rats when discriminative stimuli are compounded following separate training indicating the availability of reward after different fixed intervals (FI) on a peak procedure. One group of rats, Group Differential, learned that a flashing light stimulus signaled that one type of food pellet reward could be earned for lever pressing after an FI 5 s interval and that a second type of food pellet reward could be earned after an FI 20 s interval in the presence of a tone stimulus. A second group of rats, Group Non-Differential, was similarly trained except that both types of rewards were scheduled across flash and tone trials. When given non-reinforced flash + tone compound test trials the interval containing the maximal response rate was no different than on flash alone test trials, although some responding also appeared near the long FI time. After these FI contingencies were reversed (flash signaled FI 20 s and tone signaled FI 5 s), however, further compound test trials more clearly revealed a temporal averaging pattern in both groups. The peak interval was shifted to the right of the FI 5 stimulus. Moreover, Group Differential rats acquired the reversed discrimination somewhat more rapidly than Group Non-Differential rats, and in a final selective satiation test Group Differential rats responded less in later intervals after they had been sated on the FI 20 s reward. These data suggest that temporal averaging in stimulus compound tests occurs even when the stimuli being combined signal qualitatively different rewards, but that decreasing the value of one of those rewards can shift responding away from the relevant time interval in a selective satiation test. However, when an especially salient stimulus (e.g., flashing light) signals a short FI, rats tend to process the compound stimulus more in terms of its individual elements.

Contribution of the retrosplenial cortex to temporal discrimination learning

The retrosplenial cortex (RSC) has an important role in contextual learning and memory. While the majority of experiments have focused on the physical context, the present study asked whether the RSC is involved in processing the temporal context. Rats were trained in a temporal discrimination procedure where the duration of the intertrial interval (ITI) signaled whether or not the next tone conditioned stimulus would be paired with food pellet reinforcement. When the tone was presented after a 16-min ITI it was reinforced, but when it was presented after a 4-min ITI it was not. Rats demonstrated successful discrimination in this procedure by responding more to the tone on reinforced trials than on non-reinforced trials. Pre-training electrolytic lesions of the RSC attenuated acquisition of the temporal discrimination. The results are the first to demonstrate a role for the RSC in processing temporal information and in turn extend the role of the RSC beyond the physical context to now include the temporal context.

Asymmetry in the discrimination of length during spatial learning

The ability of rats to solve a discrimination between two objects that differ in length was investigated in five experiments. Using a rectangular swimming pool, Experiment 1 revealed it is easier to locate a submerged platform when it is near the center of a long rather than a short wall. For Experiments 2-4, the objects were black or white panels pasted onto the gray walls of a square pool, with two long panels pasted to two opposing walls and two short panels pasted to the remaining walls. The platform was easier to locate when it was placed near the middle of a long rather than a short panel. This effect was found when the long panels were twice (Experiments 2-4) or four times the length of the short panels (Experiment 4). Experiment 5 demonstrated that rats can solve a discrimination between panels of length 15 and 45 cm more readily than when they are 70 and 100 cm. The results are consistent with the claim that generalization gradients based on stimulus magnitude are steeper for stimuli that are weaker rather than stronger than the stimulus used for the original training.

Behavioral and neurobiological mechanisms of extinction in Pavlovian and instrumental learning

This article reviews research on the behavioral and neural mechanisms of extinction as it is represented in both Pavlovian and instrumental learning. In Pavlovian extinction, repeated presentation of a signal without its reinforcer weakens behavior evoked by the signal; in instrumental extinction, repeated occurrence of a voluntary action without its reinforcer weakens the strength of the action. In either case, contemporary research at both the behavioral and neural levels of analysis has been guided by a set of extinction principles that were first generated by research conducted at the behavioral level. The review discusses these principles and illustrates how they have informed the study of both Pavlovian and instrumental extinction. It shows that behavioral and neurobiological research efforts have been tightly linked and that their results are readily integrated. Pavlovian and instrumental extinction are also controlled by compatible behavioral and neural processes. Since many behavioral effects observed in extinction can be multiply determined, we suggest that the current close connection between behavioral-level and neural-level analyses will need to continue.

Separation of time-based and trial-based accounts of the partial reinforcement extinction effect

Two appetitive conditioning experiments with rats examined time-based and trial-based accounts of the partial reinforcement extinction effect (PREE). In the PREE, the loss of responding that occurs in extinction is slower when the conditioned stimulus (CS) has been paired with a reinforcer on some of its presentations (partially reinforced) instead of every presentation (continuously reinforced). According to a time-based or "time-accumulation" view (e.g., Gallistel and Gibbon, 2000), the PREE occurs because the organism has learned in partial reinforcement to expect the reinforcer after a larger amount of time has accumulated in the CS over trials. In contrast, according to a trial-based view (e.g., Capaldi, 1967), the PREE occurs because the organism has learned in partial reinforcement to expect the reinforcer after a larger number of CS presentations. Experiment 1 used a procedure that equated partially and continuously reinforced groups on their expected times to reinforcement during conditioning. A PREE was still observed. Experiment 2 then used an extinction procedure that allowed time in the CS and the number of trials to accumulate differentially through extinction. The PREE was still evident when responding was examined as a function of expected time units to the reinforcer, but was eliminated when responding was examined as a function of expected trial units to the reinforcer. There was no evidence that the animal responded according to the ratio of time accumulated during the CS in extinction over the time in the CS expected before the reinforcer. The results thus favor a trial-based account over a time-based account of extinction and the PREE. This article is part of a Special Issue entitled: Associative and Temporal Learning.

Asymmetrical generalization of conditioning and extinction from compound to element and element to compound

Four appetitive conditioning experiments studied generalization between compound conditional stimuli (AB) and their elements (e.g., A or B). In Experiments 1 and 2, rats received conditioning with A and AB, and then extinction with either A or AB. During subsequent testing, there was more generalization of extinction (nonresponding) from the compound (AB) to the element (A) than from the element (A) to the compound (AB). This asymmetry was consistent with earlier results involving temporal discrimination learning in which short and long temporal intervals played the roles of A and AB. In Experiment 3, rats received conditioning with either A or AB, and then testing with A and AB. Consistent with elemental models of conditioning, there was more generalization of conditioned responding from A to AB than from AB to A. Experiment 4 found that these asymmetries in the generalization of extinction (Experiments 1 and 2) and conditioning (Experiment 3) both contribute to the feature-positive effect. Overall, the parallel between the current findings and previous results with temporal discrimination learning supports an associative analysis of interval timing. Implications for elemental and configural theories of conditioning and generalization are also discussed.

Relapse processes after the extinction of instrumental learning: renewal, resurgence, and reacquisition

It is widely recognized that extinction (the procedure in which a Pavlovian conditioned stimulus or an instrumental action is repeatedly presented without its reinforcer) weakens behavior without erasing the original learning. Most of the experiments that support this claim have focused on several "relapse" effects that occur after Pavlovian extinction, which collectively suggest that the original learning is saved through extinction. However, although such effects do occur after instrumental extinction, they have not been explored there in as much detail. This article reviews recent research in our laboratory that has investigated three relapse effects that occur after the extinction of instrumental (operant) learning. In renewal, responding returns after extinction when the behavior is tested in a different context; in resurgence, responding recovers when a second response that has been reinforced during extinction of the first is itself put on extinction; and in rapid reacquisition, extinguished responding returns rapidly when the response is reinforced again. The results provide new insights into extinction and relapse, and are consistent with principles that have been developed to explain extinction and relapse as they occur after Pavlovian conditioning. Extinction of instrumental learning, like Pavlovian learning, involves new learning that is relatively dependent on the context for expression.

Learning and the persistence of appetite: extinction and the motivation to eat and overeat

The modern world is saturated with highly palatable and highly available food, providing many opportunities to associate food with environmental cues and actions (through Pavlovian and operant or instrumental learning, respectively). Basic learning processes can often increase the tendency to approach and consume food, whereas extinction, in which Pavlovian and operant behaviors decline when the reinforcer is withheld, weakens but does not erase those tendencies. Contemporary research suggests that extinction involves an inhibitory form of new learning that appears fragile because it is highly dependent on the context for expression. These ideas are supported by the phenomena of renewal, spontaneous recovery, resurgence, reinstatement, and rapid reacquisition in appetitive learning, which together may help explain why overeating may be difficult to suppress permanently, and why appetitive behavior may seem so persistent.

Interstimulus interval as a discriminative stimulus: evidence of the generality of a novel asymmetry in temporal discrimination learning

Three appetitive conditioning experiments with rats examined temporal discrimination learning within Pavlovian conditioning trials. In all experiments, the duration of a feature white noise stimulus signaled whether or not a subsequent 10-s target tone would be reinforced. In Experiment 1, the feature durations were 4 and 1min. For one group of rats (Group 4+/1-), 4min of noise signaled that the tone would be reinforced and 1min of noise signaled that the tone would not be reinforced. A second group (Group 1+/4-) was trained with the reverse contingency. The results showed a clear asymmetry in temporal discrimination learning: rats trained with 4+/1- (Long+/Short-) learned the discrimination readily (responding more in the tone on reinforced than on nonreinforced trials), whereas rats trained with 1+/4- (Short+/Long) did not. In Experiment 2, the feature durations were shortened to 60 and 15s. Due to strong excitatory conditioning of the 15-s feature, the reverse asymmetry was observed, with the Short+/Long- discrimination learned more readily than the Long+/Short- discrimination. However, Experiment 3 demonstrated that the original Long+/Short- advantage could be recovered while using 60- and 15-s feature durations if the excitatory conditioning of the feature was reduced by including nonreinforced feature trials. The results support previous research involving the timing of intertrial intervals and are consistent with the temporal elements hypothesis which holds that the passage of time is encoded as a series of hypothetical stimulus elements.