Temporal integration in Pavlovian appetitive conditioning in rats

Temporal scaling and computing time in neural circuits: Should we stop watching the clock and look for its gears?

Timing underlies a variety of functions, from walking to perceiving causality. Neural timing models typically fall into one of two categories-"ramping" and "population-clock" theories. According to ramping models, individual neurons track time by gradually increasing or decreasing their activity as an event approaches. To time different intervals, ramping neurons adjust their slopes, ramping steeply for short intervals and vice versa. In contrast, according to "population-clock" models, multiple neurons track time as a group, and each neuron can fire nonlinearly. As each neuron changes its rate at each point in time, a distinct pattern of activity emerges across the population. To time different intervals, the brain learns the population patterns that coincide with key events. Both model categories have empirical support. However, they often differ in plausibility when applied to certain behavioral effects. Specifically, behavioral data indicate that the timing system has a rich computational capacity, allowing observers to spontaneously compute novel intervals from previously learned ones. In population-clock theories, population patterns map to time arbitrarily, making it difficult to explain how different patterns can be computationally combined. Ramping models are viewed as more plausible, assuming upstream circuits can set the slope of ramping neurons according to a given computation. Critically, recent studies suggest that neurons with nonlinear firing profiles often scale to time different intervals-compressing for shorter intervals and stretching for longer ones. This "temporal scaling" effect has led to a hybrid-theory where, like a population-clock model, population patterns encode time, yet like a ramping neuron adjusting its slope, the speed of each neuron's firing adapts to different intervals. Here, we argue that these "relative" population-clock models are as computationally plausible as ramping theories, viewing population-speed and ramp-slope adjustments as equivalent. Therefore, we view identifying these "speed-control" circuits as a key direction for evaluating how the timing system performs computations. Furthermore, temporal scaling highlights that a key distinction between different neural models is whether they propose an absolute or relative time-representation. However, we note that several behavioral studies suggest the brain processes both scales, cautioning against a dichotomy.

Anterior cingulate neurons signal neutral cue pairings during sensory preconditioning

Of all frontocortical subregions, the anterior cingulate cortex (ACC) has perhaps the most overlapping theories of function.1-3 Recording studies in rats, humans, and other primates have reported diverse neural responses that support many theories,4-12 yet nearly all these studies have in common tasks in which one event reliably predicts another. This leaves open the possibility that ACC represents associative pairing of events, independent of their overt biological significance. Sensory preconditioning13 provides an opportunity to test this. In the first phase, preconditioning, value-neutral sensory stimuli are paired (A→B). To test whether this was learned, subjects are given standard conditioning during which one of the previously neutral sensory cues is paired with a biologically meaningful outcome (B→outcome). During the final probe test, the neutral cue which was never paired with a biologically meaningful outcome is presented alone (A→) and will elicit a conditional response, suggesting that subjects had learned the associative structure during preconditioning and use that knowledge to infer presentation of the biologically relevant outcome (A→B→outcome). Inference-based responding demonstrates a fundamental property of model-based reasoning14,15 and requires learning of the associations between neutral stimuli before rewards are introduced.16-19 ACC neurons developed firing patterns that reflected the learning of sensory associations during preconditioning, even though no rewards were present. The strength of these correlates predicted rats' ability to later mobilize and use that associative information during the probe test. These results demonstrate that clear biological significance is not necessary to produce correlates of learning in ACC.

Higher-Order Conditioning and Dopamine: Charting a Path Forward

Higher-order conditioning involves learning causal links between multiple events, which then allows one to make novel inferences. For example, observing a correlation between two events (e.g., a neighbor wearing a particular sports jersey), later helps one make new predictions based on this knowledge (e.g., the neighbor's wife's favorite sports team). This type of learning is important because it allows one to benefit maximally from previous experiences and perform adaptively in complex environments where many things are ambiguous or uncertain. Two procedures in the lab are often used to probe this kind of learning, second-order conditioning (SOC) and sensory preconditioning (SPC). In second-order conditioning (SOC), we first teach subjects that there is a relationship between a stimulus and an outcome (e.g., a tone that predicts food). Then, an additional stimulus is taught to precede the predictive stimulus (e.g., a light leads to the food-predictive tone). In sensory preconditioning (SPC), this order of training is reversed. Specifically, the two neutral stimuli (i.e., light and tone) are first paired together and then the tone is paired separately with food. Interestingly, in both SPC and SOC, humans, rodents, and even insects, and other invertebrates will later predict that both the light and tone are likely to lead to food, even though they only experienced the tone directly paired with food. While these processes are procedurally similar, a wealth of research suggests they are associatively and neurobiologically distinct. However, midbrain dopamine, a neurotransmitter long thought to facilitate basic Pavlovian conditioning in a relatively simplistic manner, appears critical for both SOC and SPC. These findings suggest dopamine may contribute to learning in ways that transcend differences in associative and neurological structure. We discuss how research demonstrating that dopamine is critical to both SOC and SPC places it at the center of more complex forms of cognition (e.g., spatial navigation and causal reasoning). Further, we suggest that these more sophisticated learning procedures, coupled with recent advances in recording and manipulating dopamine neurons, represent a new path forward in understanding dopamine's contribution to learning and cognition.

Time in Associative Learning: A Review on Temporal Maps

Ability to recall the timing of events is a crucial aspect of associative learning. Yet, traditional theories of associative learning have often overlooked the role of time in learning association and shaping the behavioral outcome. They address temporal learning as an independent and parallel process. Temporal Coding Hypothesis is an attempt to bringing together the associative and non-associative aspects of learning. This account proposes temporal maps, a representation that encodes several aspects of a learned association, but attach considerable importance to the temporal aspect. A temporal map helps an agent to make inferences about missing information by applying an integration mechanism over a common element present in independently acquired temporal maps. We review the empirical evidence demonstrating the construct of temporal maps and discuss the importance of this concept in clinical and behavioral interventions.

Chimpanzees use observed temporal directionality to learn novel causal relations

We investigated whether chimpanzees use the temporal sequence of external events to determine causation. Seventeen chimpanzees (Pan troglodytes) witnessed a human experimenter press a button in two different conditions. When she pressed the "causal button" the delivery of juice and a sound immediately followed (cause-then-effect). In contrast, she pressed the "non-causal button" only after the delivery of juice and sound (effect-then-cause). When given the opportunity to produce the desired juice delivery themselves, the chimpanzees preferentially pressed the causal button, i.e., the one that preceded the effect. Importantly, they did so in their first test trial and even though both buttons were equally associated with juice delivery. This outcome suggests that chimpanzees, like human children, do not rely solely on their own actions to make use of novel causal relations, but they can learn causal sequences based on observation alone. We discuss these findings in relation to the literature on causal inferences as well as associative learning.

Spontaneous integration of temporal information: implications for representational/computational capacity of animals

How do animals adapt their behaviors to changing conditions? This question relates to the debate between associative versus representational/computational approaches in cognitive science. An influential line of research that has significantly shaped the conceptual development of animal learning over decades has primarily focused on the role of associative dynamics with little-to-no ascription of representational/combinatorial capacities. The common assumption of these models is that behavioral adjustments are incremental and they result from updating of associations based on actions and their outcomes, without encoding the critical information serving as the determinant(s) of such contingencies (e.g., time in interval schedules, number in ratio schedules). On the other hand, an independent line of research provides evidence for behavioral phenomena that cannot be readily accounted for by the conventional associationist approach. In this paper, we will review different sets of findings particularly in the area of interval timing that suggest the ability of animals to make swift spontaneous computations on subjective quantities and incorporate them into their behavior. Findings of these studies constitute empirical challenges for the associationist approaches to behavioral flexibility. We argue that interval timing is a fertile ground for the formulation of critical tests of different theoretical approaches to animal behavior.

Temporal integration and instrumental conditioned reinforcement

Stimuli associated with primary reinforcement for instrumental behavior are widely believed to acquire the capacity to function as conditioned reinforcers via Pavlovian conditioning. Some Pavlovian conditioning studies suggest that animals learn the important temporal relations between stimuli and integrate such temporal information over separate experiences to form a temporal map. The present experiment examined whether Pavlovian conditioning can establish a positive instrumental conditioned reinforcer through such temporal integration. Two groups of rats received either delay or trace appetitive conditioning in which a neutral stimulus predicted response-independent food deliveries (CS1→US). Both groups then experienced one session of backward second-order conditioning of the training CS1 and a novel CS2 (CS1-CS2 pairing). Finally, the ability of CS2 to function as a conditioned reinforcer for a new instrumental response (leverpressing) was assessed. Consistent with the previous demonstrations of temporal integration in fear conditioning, a CS2 previously trained in a trace-conditioning protocol served as a better instrumental conditioned reinforcer after backward second-order conditioning than did a CS2 previously trained in a delay protocol. These results suggest that an instrumental conditioned reinforcer can be established via temporal integration and raise challenges for existing quantitative accounts of instrumental conditioned reinforcement.

The effect of temporal information among events on Bayesian causal inference in rats

A temporal relationship between events of potential cause and effect is critical to generate a causal relationship because the cause has to be followed by the effect. The present study investigated the role of temporal relationships between events in causal inference in rats via Pavlovian pairings. In Experiment 1A, subjects in Group Successive received training trials whereby Event 1 (tone or light) was followed by Events 2 (light or tone) and 3 (sucrose solution), whereas those in Group Simultaneous received simultaneous pairings of Events 1 and 2, and Events 1 and 3. During testing, a lever was inserted into the experimental chamber, where subjects were allowed to press the lever which produced the occurrence of Event 2 without reward. By measuring nose-poke responses during the presentation of Event 2, assumingly based on the prediction of occurrence of sucrose solution, subjects in Group Successive showed a relatively lower response rate than did those in Group Simultaneous. In Experiment 1B, this difference was not observed if subjects received the presentations of Event 2 which was irrelevant to their lever pressing during testing. These results suggest that rats can differentiate their response based on the elemental temporal information even when the integrated temporal map was the same, and implied that rats use temporal information as well as conditional probability based on causal Bayesian network account.

Spatial integration of boundaries in a 3D virtual environment

Prior research, using two- and three-dimensional environments, has found that when both human and nonhuman animals independently acquire two associations between landmarks with a common landmark (e.g., LM1-LM2 and LM2-LM3), each with its own spatial relationship, they behave as if the two unique LMs have a known spatial relationship despite their never having been paired. Seemingly, they have integrated the two associations to create a third association with its own spatial relationship (LM1-LM3). Using sensory preconditioning (Experiment 1) and second-order conditioning (Experiment 2) procedures, we found that human participants integrated information about the boundaries of pathways to locate a goal within a three-dimensional virtual environment in the absence of any relevant landmarks. Spatial integration depended on the participant experiencing a common boundary feature with which to link the pathways. These results suggest that the principles of associative learning also apply to the boundaries of an environment.

Temporal maps in appetitive Pavlovian conditioning

Previous research suggests animals may integrate temporal information into mental representations, or temporal maps. We examined the parameters under which animals integrate temporal information in three appetitive conditioning experiments. In Experiment 1 the temporal relationship between 2 auditory cues was established during sensory preconditioning (SPC). Subsequently, rats were given first order conditioning (FOC) with one of the cues. Results showed integration of the order of cues between the SPC and FOC training phases. In subsequent experiments we tested the hypothesis that quantitative temporal information can be integrated across phases. In Experiment 2, SPC of two short auditory cues superimposed on a longer auditory cue was followed by FOC of either one of the short cues, or of the long cue at different times in the cue. Contrary to our predictions we did not find evidence of integration of temporal information across the phases of the experiment and instead responding to the SPC cues in Experiment 2 appeared to be dominated by generalization from the FOC cues. In Experiment 3 shorter auditory cues were superimposed on a longer duration light cue but with asynchronous onset and offset of the superimposed cues. There is some evidence consistent with the hypothesis that quantitative discrimination of whether reward should be expected during the early or later parts of a cue could be integrated across experiences. However, the pattern of responding within cues was not indicative of integration of quantitative temporal information. Generalization of expected times of reward during FOC seems to be the dominant determinant of within-cue response patterns in these experiments. Consequently, while we clearly demonstrated the integration of temporal order in the modulation of this dominant pattern we did not find strong evidence of integration of precise quantitative temporal information. This article is part of a Special Issue entitled: Associative and Temporal Learning.

Damage to posterior parietal cortex impairs two forms of relational learning

The posterior parietal cortex (PPC) is a component of a major cortico-hippocampal circuit that is involved in relational learning, yet the specific contribution of PPC to hippocampal-dependent learning is unresolved. To address this, two experiments were carried out to test the effects of PPC damage on tasks that involve forming associations between multiple sensory stimuli. In Experiment 1, sham or electrolytic lesions of the PPC were made before rats were tested on a three-phase sensory preconditioning task. During the first phase, half of the training trials consisted of pairings of an auditory stimulus followed by a light. During the other trials, a second auditory stimulus was presented alone. In the next phase of training, the same light was paired with food, but no auditory stimuli were presented. During the final phase of the procedure both auditory stimuli were presented in the absence of reinforcement during a single test session. As is typically observed during the test session, control rats exhibited greater conditioned responding to the auditory cue that was previously paired with light compared to the unpaired cue. In contrast, PPC-lesioned rats responded equally to both auditory cues. In Experiment 2, PPC-lesioned and control rats were trained in a compound feature negative discrimination task consisting of reinforced presentations of a tone-alone and non-reinforced simultaneous presentations of a light-tone compound stimulus. Control rats but not rats with damage to the PPC successfully learned the discrimination. Collectively, these results support the idea that the PPC contributes to relational learning involving multimodal sensory stimuli, perhaps by regulating the attentional processing of conditioned stimuli.

Factors that influence negative summation in a spatial-search task with pigeons

A variant of the standard conditioned inhibition procedure was used to evaluate landmark-based spatial search in a touchscreen preparation. Pigeons were given compound trials with one landmark (A) positioned in a consistent spatial relationship to a hidden goal and another landmark (B) positioned randomly with respect to A and the hidden goal (AB+). On half of the non-reinforced inhibitory trials, A was paired with landmark X (AX-) and on the remaining trials B was paired with Y (BY-). All subjects were also given reinforced trials with a transfer excitor (T+). During conditioned inhibition training, subjects showed no change in overall responding during AX- trials but did show a decrease in the number of pecks to the goal location signaled by A. During non-reinforced summation tests with landmark T, X had a greater suppressive effect than did Y on overall responding but the percentage of pecks at the goal did not differ unless X was positioned near the expected goal signaled by T. These data demonstrate that the effectiveness of a stimulus trained as an inhibitor is dependent on the strength of the association between its training excitor (A) and the US, as well as, the spatial arrangement of stimuli during testing.

The Modulation of Operant Variation by the Probability, Magnitude, and Delay of Reinforcement

Recent studies have demonstrated that the expectation of reward delivery has an inverse relationship with operant behavioral variation (e.g., Stahlman, Roberts, & Blaisdell, 2010). Research thus far has largely focused on one aspect of reinforcement - the likelihood of food delivery. In two experiments with pigeons, we examined the effect of two other aspects of reinforcement: the magnitude of the reward and the temporal delay between the operant response and outcome delivery. In the first experiment, we found that a large reward magnitude resulted in reduced spatiotemporal variation in pigeons' pecking behavior. In the second experiment, we found that a 4-s delay between response-dependent trial termination and reward delivery increased variation in behavior. These results indicate that multiple dimensions of the reinforcer modulate operant response variation.

Design of a neurally plausible model of fear learning

A neurally oriented conceptual and computational model of fear conditioning manifested by freezing behavior (FRAT), which accounts for many aspects of delay and context conditioning, has been constructed. Conditioning and extinction are the result of neuromodulation-controlled LTP at synapses of thalamic, cortical, and hippocampal afferents on principal cells and inhibitory interneurons of lateral and basal amygdala. The phenomena accounted for by the model (and simulated by the computational version) include conditioning, secondary reinforcement, blocking, the immediate shock deficit, extinction, renewal, and a range of empirically valid effects of pre- and post-training ablation or inactivation of hippocampus or amygdala nuclei.

Temporal factors control hippocampal contributions to fear renewal after extinction

Fear can be extinguished by repeated exposure to a cue that signals threat. However, extinction does not erase fear, as an extinguished cue presented in a context distinct from that of extinction results in renewed fear of that cue. The hippocampus, which is involved in the formation of contextual representations, is a natural candidate structure for investigations into the neural circuitry underlying fear renewal. Thus far, studies examining the necessity of the hippocampus for fear renewal have produced mixed results. We isolated the conditions under which the hippocampus may be required for renewal. Rats received lesions of the dorsal hippocampus either prior to tone fear conditioning or following extinction. Fear renewal was measured using discrete tone presentations or a long, continuous tone. The topography of fear responding at test was assessed by comparing "early" and "sustained" renewal, where early fear was determined by freezing to the first discrete tone or the equivalent initial segment of a continuous tone and sustained fear was determined by freezing averaged across all discrete tones or the entire continuous tone. We found that following pretraining damage of the hippocampus, early renewal remained intact regardless of lesion condition. However, sustained renewal only persisted in discrete, but not continuous, tone-tested animals. A more extensive analysis of the topography of fear responding revealed that the disruption of renewal was generated when the tone duration at test began to violate that used during extinction, suggesting that the hippocampus is sensitive to mismatches in CS-duration. Postextinction lesions resulted in an overall reduction of fear renewal. This pattern of results is consistent with those observed for contextual fear conditioning, wherein animals display a resistance to anterograde amnesia despite the presence of a strong retrograde amnesia for the same contextual information. Furthermore, the data support a role for the hippocampus in sustaining renewal when the CS duration at test does not match that used during extinction.

Integration of spatial relationships and temporal relationships in humans

Three experiments tested human participants on a two-dimensional, computer, landmark-based search task to assess the integration of independently acquired spatial and temporal relationships. Experiment 1 showed that A-B spatial training followed by B-outcome spatial training resulted in spatial integration in such a way that A was effectively associated with the outcome. Experiment 2 showed that A-B spatial and temporal training followed by B-outcome spatial and temporal training resulted in integration that created both spatial and temporal relationships between A and the outcome. Experiment 3 refuted an alternative explanation, one that is based on decision-making speed, to the temporal-integration strategy that was suggested by Experiment 2. These results replicate in humans the observations regarding spatial integration made by Sawa, Leising, and Blaisdell (2005) using a spatial-search task with pigeons, and they extend those observations to temporal integration.

Time and Associative Learning

In a basic associative learning paradigm, learning is said to have occurred when the conditioned stimulus evokes an anticipatory response. This learning is widely believed to depend on the contiguous presentation of conditioned and unconditioned stimulus. However, what it means to be contiguous has not been rigorously defined. Here we examine the empirical bases for these beliefs and suggest an alternative view based on the hypothesis that learning about the temporal relationships between events determines the speed of emergence, vigor and form of conditioned behavior. This temporal learning occurs very rapidly and prior to the appearance of the anticipatory response. The temporal relations are learned even when no anticipatory response is evoked. The speed with which an anticipatory response emerges is proportional to the informativeness of the predictive cue (CS) regarding the rate of occurrence of the predicted event (US). This analysis gives an account of what we mean by "temporal pairing" and is in accord with the data on speed of acquisition and basic findings in the cue competition literature. In this account, learning depends on perceiving and encoding temporal regularities rather than stimulus contiguities.

Feature-short and feature-long discrimination learning in the pigeon: conditional control of a two-event temporal map

Eight pigeons were trained to peck an illuminated target key on discrete-trial fixed-interval schedules of reinforcement by food. Four birds were exposed to a feature-short (FS) task where a feature light signaled shortening of the forthcoming target-outcome interval from 30 to 15s, while the other four birds were exposed to a feature-long (FL) task where a feature light signaled extension of the forthcoming target-outcome interval from 15 to 30s. The discrimination performance measured by differential temporal distributions of pecks between featured and non-featured target trials suggested that the target-food temporal map was under conditional control of the feature light in both groups. The FS discrimination was more difficult to learn than the FL discrimination. This FS inferiority implies that our birds did not resort on the simple temporal discrimination by timing from the trial onset. The simple temporal discrimination account was also negated by the finding that increasing the feature-target gap did not have a predicted effect on the response distribution.