Are separate theories of conditioning and timing necessary?

Learning About Reward Identities and Time

We discuss three empirical findings that we think any theory attempting to integrate interval timing with associative learning concepts will need to address. These empirical phenomena all come from studies that combine peak timing procedures with reinforcer devaluation or conditional discrimination tasks commonly employed, respectively, in interval timing or associative learning research traditions. The three phenomena we discuss include: (1) the observation that disruptions in reward identity encoding have little to no impact on the encoding of reward time in the a peak procedure (Delamater., 1998), (2) the findings that organisms tend to average their time estimates when presented with a stimulus compound consisting of separately learned stimuli indicating short or long reward times but that such temporal averaging, itself, is sensitive to post-conditioning selective reward devaluation, and (3) that rats can learn a temporal patterning task in which two stimuli presented independently indicate one time to reward availability while their compound indicates another. We review our prior results and present new findings illustrating these three phenomena and we discuss the special challenges they pose for cascade theories of timing, for multiple-oscillator models, and for any approach that attempts to integrate interval timing and associative models. We close by illustrating some ways in which multi-layer connectionist network models might begin to address some of our key findings. We believe this will require an approach that includes separate mechanisms that code for reward identity and time, but that does so in a way that permits for integration between the two systems.

The effect of reinforcement probability on time discrimination in the midsession reversal task

We examined how biasing time perception affects choice in a midsession reversal task. Given a simultaneous discrimination between stimuli S1 and S2, with choices of S1 reinforced during the first, but not the second half of the trials, and choices of S2 reinforced during the second, but not the first half of the trials, pigeons show anticipation errors (premature choices of S2) and perseveration errors (belated choices of S1). This suggests that choice depends on timing processes, on predicting when the contingency reverses based on session duration. We exposed 7 pigeons to a midsession reversal task and manipulated the reinforcement rate on each half of the session. Compared to equal reinforcement rates on both halves of the session, when the reinforcement rate on the first half was lower than on the second half, performance showed more anticipation and less perseveration errors, and when the reinforcement rate on the first half was higher than on the second half, performance showed a remarkable reduction of both types of errors. These results suggest that choice depends on both time into the session and the outcome of previous trials. They also challenge current models of timing to integrate local effects.

Learning about the CS during latent inhibition: Preexposure enhances temporal control

In 3 experiments, rats were given nonreinforced preexposure to an auditory stimulus, after which this stimulus and a second, novel cue were paired with food. Lower rates of conditioned responding were observed to the preexposed stimulus across the 3 experiments, indicative of latent inhibition. The degree to which animals used these cues to time the occurrence of food delivery was also examined. Paradoxically, the response slopes—indicating the rate of increase in responding over the course of the conditioned stimulus—were greater for the preexposed than for the novel cues, consistent with the suggestion that the preexposed stimulus exerted greater temporal control. Moreover, this was the case irrespective of whether the duration of the cue during preexposure differed from that during conditioning. These results suggest that although conditioned stimulus preexposure retards conditioning, it may enhance timing. The findings are discussed in terms of current models of conditioning and timing.

Everywhere and everything: The power and ubiquity of time

Anticipatory timing plays a critical role in many aspects of human and non-human animal behavior. Timing has been consistently observed in the range of milliseconds to hours, and demonstrates a powerful influence on the organization of behavior. Anticipatory timing is acquired early in associative learning and appears to guide association formation in important ways. Importantly, timing participates in regulating goal-directed behaviors in many schedules of reinforcements, and plays a critical role in value-based decision making under concurrent schedules. In addition to playing a key role in fundamental learning processes, timing often dominates when temporal cues are available concurrently with other stimulus dimensions. Such control by the passage of time has even been observed when other cues provide more accurate information and can lead to sub-optimal behaviors. The dominance of temporal cues in governing anticipatory behavior suggests that time may be inherently more salient than many other stimulus dimensions. Discussions of the interface of the timing system with other cognitive processes are provided to demonstrate the powerful and primitive nature of time as a stimulus dimension.

Overshadowing by fixed- and variable-duration stimuli

Two experiments investigated the effect of the temporal distribution form of a stimulus on its ability to produce an overshadowing effect. The overshadowing stimuli were either of the same duration on every trial, or of a variable duration drawn from an exponential distribution with the same mean duration as that of the fixed stimulus. Both experiments provided evidence that a variable-duration stimulus was less effective than a fixed-duration cue at overshadowing conditioning to a target conditioned stimulus (CS); moreover, this effect was independent of whether the overshadowed CS was fixed or variable. The findings presented here are consistent with the idea that the strength of the association between CS and unconditioned stimulus (US) is, in part, determined by the temporal distribution form of the CS. These results are discussed in terms of time-accumulation and trial-based theories of conditioning and timing.

Effect of distracter preexposure on the reset of an internal clock

Interruptions and unfamiliar events (distracters) during a timed signal disrupt (delay) timing in humans and other animals. We hypothesized that repeated exposure to a stimulus may reduce its subsequent time-disrupting properties. To test this hypothesis rats were trained in a reversed peak-interval (RPI) procedure, in which dark timing trials were separated by illuminated inter-trial intervals. Rats were then repeatedly exposed to an auditory stimulus (noise) in either dark (DARK group), or illuminated chambers (LIGHT group); control rats were not exposed to the noise (NOVEL group). Afterwards, the time-resetting properties of the noise were tested by presenting it unexpectedly during the (dark) RPI trials. The noise reset timing in NOVEL rats, but stopped timing in DARK rats, suggesting that preexposure reduces the time-resetting effects of distracters. However, in LIGHT rats, the noise stopped timing when the presented early in the RPI trial, but reset when presented late, suggesting that exposure to noise was only partly effective in overriding other relevant variables, such as distracter location. These results suggest that the effect of distracter preexposure on the reset of an internal clock depends on complex associative and temporal interactions which require further investigations. This article is part of a Special Issue entitled: Associative and Temporal Learning.

Associative and temporal processes: a dual process approach

Approaches to the study of associative learning and interval timing have traditionally diverged on methodological and theoretical levels of analysis. However, more recent attempts have been made to explain one class of phenomena in terms of the other using various single-process approaches. In this paper we suggest that an interactive dual-process approach might more accurately reflect underlying behavioral and neural processes. We will argue that timing in Pavlovian conditioning is best understood in terms of an abstract temporal code that is not a feature of the predictive stimulus (i.e., the conditioned stimulus, CS), per se. Rather, we assume that the time between the CS and the unconditioned stimulus (US) is encoded in the form of an abstract representation of this temporal interval produced as an output of a central multiple-oscillator interval timing system. As such, associations can then develop between the CS and this abstract temporal code in much the same way that the CS develops associations with different features of the US. To support the dual-process approach, we first show that exposure to a Pavlovian zero contingency procedure results in a failure to acquire new associations, not a failure to express learning due to some temporally defined performance mask. We also consider evidence that supports the abstract temporal coding idea in a US preexposure task, and, finally, present some evidence to encourage the dissociation between basic associative and temporal learning processes by exploring reward devaluation effects in a peak timing task.

Interactions of timing and prediction error learning

Timing and prediction error learning have historically been treated as independent processes, but growing evidence has indicated that they are not orthogonal. Timing emerges at the earliest time point when conditioned responses are observed, and temporal variables modulate prediction error learning in both simple conditioning and cue competition paradigms. In addition, prediction errors, through changes in reward magnitude or value alter timing of behavior. Thus, there appears to be a bi-directional interaction between timing and prediction error learning. Modern theories have attempted to integrate the two processes with mixed success. A neurocomputational approach to theory development is espoused, which draws on neurobiological evidence to guide and constrain computational model development. Heuristics for future model development are presented with the goal of sparking new approaches to theory development in the timing and prediction error fields.

Timing and cue competition in conditioning of the nictitating membrane response of the rabbit (Oryctolagus cuniculus)

Rabbits were classically conditioned using compounds of tone and light conditioned stimuli (CSs) presented with either simultaneous onsets (Experiment 1) or serial onsets (Experiment 2) in a delay conditioning paradigm. Training with the simultaneous compound reduced the likelihood of a conditioned response (CR) to the individual CSs (“mutual overshadowing”) but left CR timing unaltered. CR peaks were consistently clustered around the time of unconditioned stimulus (US) delivery. Training with the serial compound (CSA→CSB→US) reduced responding to CSB (“temporal primacy/information effect”) but this effect was prevented by prior CSB→US pairings. In both cases, serial compound training altered CR timing. On CSA→CSB test trials, the CRs were accelerated; the CR peaks occurred after CSB onset but well before the time of US delivery. Conversely, CRs on CSB– trials were decelerated; the distribution of CR peaks was variable but centered well after the US. Timing on CSB– trials was at most only slightly accelerated. The results are discussed with respect to processes of generalization and spectral timing applicable to the cerebellar and forebrain pathways in eyeblink preparations.

Acquisition of "Start" and "Stop" response thresholds in peak-interval timing is differentially sensitive to protein synthesis inhibition in the dorsal and ventral striatum

Time-based decision-making in peak-interval timing procedures involves the setting of response thresholds for the initiation (“Start”) and termination (“Stop”) of a response sequence that is centered on a target duration. Using intracerebral infusions of the protein synthesis inhibitor anisomycin, we report that the acquisition of the “Start” response depends on normal functioning (including protein synthesis) in the dorsal striatum (DS), but not the ventral striatum (VS). Conversely, disruption of the VS, but not the DS, impairs the acquisition of the “Stop” response. We hypothesize that the dorsal and ventral regions of the striatum function as a competitive neural network that encodes the temporal boundaries marking the beginning and end of a timed response sequence.

Timing in trace conditioning of the nictitating membrane response of the rabbit (Oryctolagus cuniculus): scalar, nonscalar, and adaptive features

Using interstimulus intervals (ISIs) of 125, 250, and 500 msec in trace conditioning of the rabbit nictitating membrane response, the offset times and durations of conditioned responses (CRs) were collected along with onset and peak latencies. All measures were proportional to the ISI, but only onset and peak latencies conformed to the criterion for scalar timing. Regarding the CR's possible protective overlap of the unconditioned stimulus (US), CR duration increased with ISI, while the peak's alignment with the US declined. Implications for models of timing and CR adaptiveness are discussed.

Learning to Time: a perspective

In the last decades, researchers have proposed a large number of theoretical models of timing. These models make different assumptions concerning how animals learn to time events and how such learning is represented in memory. However, few studies have examined these different assumptions either empirically or conceptually. For knowledge to accumulate, variation in theoretical models must be accompanied by selection of models and model ideas. To that end, we review two timing models, Scalar Expectancy Theory (SET), the dominant model in the field, and the Learning-to-Time (LeT) model, one of the few models dealing explicitly with learning. In the first part of this article, we describe how each model works in prototypical concurrent and retrospective timing tasks, identify their structural similarities, and classify their differences concerning temporal learning and memory. In the second part, we review a series of studies that examined these differences and conclude that both the memory structure postulated by SET and the state dynamics postulated by LeT are probably incorrect. In the third part, we propose a hybrid model that may improve on its parents. The hybrid model accounts for the typical findings in fixed-interval schedules, the peak procedure, mixed fixed interval schedules, simple and double temporal bisection, and temporal generalization tasks. In the fourth and last part, we identify seven challenges that any timing model must meet.

Context effects in a temporal discrimination task" further tests of the Scalar Expectancy Theory and Learning-to-Time models

Pigeons were trained on two temporal bisection tasks, which alternated every two sessions. In the first task, they learned to choose a red key after a 1-s signal and a green key after a 4-s signal; in the second task, they learned to choose a blue key after a 4-s signal and a yellow key after a 16-s signal. Then the pigeons were exposed to a series of test trials in order to contrast two timing models, Learning-to-Time (LeT) and Scalar Expectancy Theory (SET). The models made substantially different predictions particularly for the test trials in which the sample duration ranged from 1 s to 16 s and the choice keys were Green and Blue, the keys associated with the same 4-s samples: LeT predicted that preference for Green should increase with sample duration, a context effect, but SET predicted that preference for Green should not vary with sample duration. The results were consistent with LeT. The present study adds to the literature the finding that the context effect occurs even when the two basic discriminations are never combined in the same session.

The influence of CS-US interval on several different indices of learning in appetitive conditioning

Four experiments examined the effects of varying the conditioned stimulus-unconditioned stimulus (CS-US) interval (and US density) on learning in an appetitive magazine approach task with rats. Learning was assessed with conditioned response (CR) measures, as well as measures of sensory-specific stimulus-outcome associations (Pavlovian-instrumental transfer, potentiated feeding, and US devaluation). The results from these studies indicate that there exists an inverse relation between CS-US interval and magazine approach CRs, but that sensory-specific stimulus-outcome associations are established over a wide range of relatively long, but not short, CS-US intervals. These data suggest that simple CR measures provide different information about what is learned than measures of the specific stimulus-outcome association, and that time is a more critical variable for the former than latter component of learning.

Further tests of the Scalar Expectancy Theory (SET) and the Learning-to-Time (LeT) model in a temporal bisection task

To contrast two models of timing, Scalar Expectancy Theory (SET) and Learning to Time (LeT), pigeons were exposed to a double temporal bisection procedure. On half of the trials, they learned to choose a red key after a 1s signal and a green key after a 4s signal; on the other half of the trials, they learned to choose a blue key after a 4-s signal and a yellow key after a 16-s signal. This was Phase A of an ABA design. On Phase B, the pigeons were divided into two groups and exposed to a new bisection task in which the signals ranged from 1 to 16s and the choice keys were blue and green. One group was reinforced for choosing blue after 1-s signals and green after 16-s signals and the other group was reinforced for the opposite mapping (green after 1-s signals and blue after 16-s signals). Whereas SET predicted no differences between the groups, LeT predicted that the former group would learn the new discrimination faster than the latter group. The results were consistent with LeT. Finally, the pigeons returned to Phase A. Only LeT made specific predictions regarding the reacquisition of the four temporal discriminations. These predictions were only partly consistent with the results.

Temporal learning in random control procedures

Experiments 1 and 2 delivered conditioned stimuli (CSs) at random times and unconditioned stimuli (USs) at either fixed (Experiment 1) or random (Experiment 2) intervals. In Experiment 3, CS duration was manipulated, and US deliveries occurred at random during the background. In all 3 experiments, the mean rate of responding (head entries into the food cup) in the background was determined by the mean US-US interval, and the mean rate during the CS was a linear combination of responding controlled by the mean US-US and mean CS onset-US intervals; the pattern of responding in time was determined by the interval distribution form (fixed or random). An event-based timing account, Packet theory, provided an explanation of the results.

Stimulus representation in SOP: II. An application to inhibition of delay

The componential extension of SOP accounts for conditioned response (CR) timing in Pavlovian conditioning by assuming that learning accrues with relative independence to stimulus elements that are differentially occasioned during the duration of the conditioned stimulus (CS). SOP, using a competitive learning rule and the assumption that temporal learning emerges via resolution of what is equivalent to an "AX+BX-" discrimination, predicts a progressive increase in the latency of the CR over training, or what Pavlov refer to as "inhibition of delay." Other componential models, which use noncompetitive learning rules, do not predict inhibition of delay. Either type of model makes the prediction indicated, independently of the length of the CS-unconditioned stimulus (US) interval. We report two experiments that demonstrated inhibition of delay when rabbits were trained with relatively long, but not with short, CS-US intervals. To account for this divergence, we assumed that the SOP stimulus trace involves two kinds of elements, some with a temporally distributed pattern of activity over the duration of the CS duration, and some with a randomly distributed pattern. This stimulus representation, not only allows for inhibition of delay with long but not short CS-US intervals, but in combination with SOP's performance rule deduces CR's with "Weber variability."

Packet theory of conditioning and timing

Packet theory is based on the assumption that the momentary probability of producing a bout or packet of responding is controlled by the conditional expected time function. Bouts of head entry responses of rats into a food cup appear to have the same characteristics across a range of conditions. The conditional expected time function is the mean expected time remaining until the next food delivery as a function of time since an event such as food or stimulus onset. The conditional expected time function encodes mean interval duration as well as the distribution form so that both the mean response rate and form of responding in time can be predicted. Simulations of Packet theory produced accurate quantitative predictions of: (1) the effect of reinforcement density (mean food-food interval) and distribution form on responding; (2) scalar variance in fixed interval responding; (3) CS-US and intertrial interval effects on the strength of conditioning; and (4) the effect of the ratio of cycle:trial time on the strength of conditioning.

Shifts in the psychometric function and their implications for models of timing

This study examined how two models of timing, scalar expectancy theory (SET) and learning to time (LeT), conceptualize the learning process in temporal tasks, and then reports two experiments to test these conceptualizations. Pigeons responded on a two-alternative free-operant psychophysical procedure in which responses on the left key were reinforceable during the first two, but not the last two, quarters of a 60-s trial, and responses on the right key were reinforceable during the last two, but not the first two, quarters of the trial. In Experiment 1 three groups of birds experienced a difference in reinforcement rates between the two keys only at the end segments of the trial (i.e., between the first and fourth quarters), only around the middle segments of the trial (i.e., between the second and third quarters), or in both end and middle segments. In Condition 1 the difference in reinforcement rate favored the left key; in Condition 2 it favored the right key. When the reinforcement rates differed in the end segments of the trial, the psychometric function--the proportion of right responses across the trial--did not shift across conditions; when it occurred around the middle of the trial or in both end and middle segments, the psychometric function shifted across conditions. Experiment 2 showed that the psychometric function shifts even when the overall reinforcement rate for the two keys is equal, provided the rates differ around the middle of the trial. This pattern of shifts of the psychometric function is inconsistent with SET. In contrast, LeT provided a good quantitative fit to the data.