Overexpectation in appetitive Pavlovian and instrumental conditioning

Use of the overexpectation effect to reduce conditioned seeking behavior controlled by nicotine

Nicotine produces robust stimulus effects that can be conditioned to form associations with reinforcing nondrug stimuli. We examine how established associations to the nicotine stimulus may be weakened via the overexpectation effect. In two experiments, we separately conditioned sucrose associations to the interoceptive nicotine stimulus (0.4 mg/kg, SC) and to a "noisy" exteroceptive contextual stimulus (oscillating houselight and white noise) via the discriminated goal-tracking task. Thereafter, we presented additional sucrose pairings with the nicotine and noisy stimuli, now in compound. Testing of the conditioned goal-tracking evoked by the nicotine and noisy stimuli in isolation-before versus after compound conditioning (Experiment 1) or between treatment and control groups (Experiment 2)-demonstrated an attenuation of conditioned responding via the overexpectation effect. We suggest that applications of the overexpectation effect may provide some promise for treatments seeking to attenuate drug-evoked conditioned responses in situations where extinction-based interventions are not suitable.

Elevated fear states facilitate ventral hippocampal engagement of basolateral amygdala neuronal activity

Fear memory formation and retention rely on the activation of distributed neural circuits. The basolateral amygdala (BLA) and ventral hippocampus (VH) in particular are two regions that support contextual fear memory processes and share reciprocal connections. The VH → BLA pathway is critical for increases in fear after initial learning, in both fear renewal following extinction learning and during fear generalization. This raises the possibility that functional changes in VH projections to the BLA support increases in learned fear. In line with this, fear can also be increased with alterations to the original content of the memory via reconsolidation, as in fear elevation procedures. However, very little is known about the functional changes in the VH → BLA pathway supporting reconsolidation-related increases in fear. In this study, we used in vivo extracellular electrophysiology to examine the functional neuronal changes within the BLA and in the VH → BLA pathway as a result of fear elevation and standard fear retrieval procedures. Elevated fear expression was accompanied by higher BLA spontaneous firing compared to a standard fear retrieval condition. Across a range of stimulation frequencies, we also found that VH stimulation evoked higher BLA firing following fear elevation compared to standard retrieval. These results suggest that fear elevation is associated with an increased capacity of the VH to drive neuronal activity in the BLA, highlighting a potential circuit involved in strengthening existing fear memories.

Learning to stop responding

Learning to stop responding is an important process that allows behavior to adapt to a changing and variable environment. This article reviews recent research in this laboratory and others that has studied how animals learn to stop responding in operant extinction, punishment, and feature-negative learning. Extinction and punishment are shown to be similar in two fundamental ways. First, the response-suppressing effects of both are highly context-specific. Second, the response-suppressing effects of both can be remarkably response-specific: Inhibition of one response transfers little to other responses. Learning to inhibit the response so specifically may result from the correction of "response error," the difference between the level of responding and what the current reinforcer supports. In contrast, the inhibition of responding that develops in feature-negative learning, where the response is reinforced during one discriminative stimulus (A) but not in a compound of A and stimulus B, is less response-specific: The inhibition of responding by stimulus B transfers and inhibits a second response, especially if the second response has itself been inhibited before. The results thus indicate both response-specific and response-general forms of behavioral inhibition. One possibility is that response-specific inhibition is learned when the circumstances encourage the organism to pay attention to the response-to what it is actually doing-as behavioral suppression is learned.

Spontaneous recovery from overexpectation in an insect

In associative learning in mammals, it is widely accepted that learning is determined by the prediction error, i.e., the error between the actual reward and the reward predicted by the animal. However, it is unclear whether error-based learning theories are applicable to the learning occurring in other non-mammalian species. Here, we examined whether overexpectation, a phenomenon that supports error-based learning theories, occurs in crickets. Crickets were independently trained with two different conditioned stimuli (CSs), an odour and a visual pattern, that were followed by an appetitive unconditioned stimulus (US). Then the two CSs were presented simultaneously as a compound, followed by the same US. This treatment resulted in a reduced conditioned response to the odour CS when tested immediately after training. However, the response to the CS was partially recovered after 1 day. These results are the first to show overexpectation and its spontaneous recovery in an invertebrate species. While the results showing overexpectation are in agreement with the prediction by the Rescorla-Wagner model, a major form of error-based learning theories, the ones showing spontaneous recovery are not. Our results suggest that conventional error-based learning models account for some, but not for all essential features of Pavlovian conditioning in crickets.

BEHAVIORAL AND NEUROBIOLOGICAL MECHANISMS OF PAVLOVIAN AND INSTRUMENTAL EXTINCTION LEARNING

This article reviews the behavioral neuroscience of extinction, the phenomenon in which a behavior that has been acquired through Pavlovian or instrumental (operant) learning decreases in strength when the outcome that reinforced it is removed. Behavioral research indicates that neither Pavlovian nor operant extinction depends substantially on erasure of the original learning but instead depends on new inhibitory learning that is primarily expressed in the context in which it is learned, as exemplified by the renewal effect. Although the nature of the inhibition may differ in Pavlovian and operant extinction, in either case the decline in responding may depend on both generalization decrement and the correction of prediction error. At the neural level, Pavlovian extinction requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in extinction retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. Instrumental extinction involves distinct ensembles in corticostriatal, striatopallidal, and striatohypothalamic circuits as well as their thalamic returns for inhibitory (extinction) and excitatory (renewal and other relapse phenomena) control over operant responding. The field has made significant progress in recent decades, although a fully integrated biobehavioral understanding still awaits.

Numerical averaging in mice

Rodents can be trained to associate different durations with different stimuli (e.g., light/sound). When the associated stimuli are presented together, maximal responding is observed around the average of individual durations (akin to averaging). The current study investigated whether mice can also average independently trained numerosities. Mice were initially trained to make 10 or 20 lever presses on a single (run) lever to obtain a reward and each fixed-ratio schedule was signaled either with an auditory or visual stimulus. Then, mice were trained to press another lever to obtain the reward after they responded on the run lever for the minimum number of presses [Fixed Consecutive Number (FCN)-10 or -20 trials] signaled by the corresponding discriminative stimulus. Following this training, FCN trials with the compound stimulus were introduced to test the counting behavior of mice when they encountered conflicting information regarding the number of responses required to obtain the reward. Our results showed that the numbers of responses on these compound test trials were around the average of the number of responses in FCN-10 and FCN-20 trials particularly when the auditory stimulus was associated with a fewer number of required responses. The counting strategy explained the behavior of the majority of the mice in the FCN-Compound test trials (as opposed to the timing strategy). The number of responses in FCN-Compound trials was accounted for equally well by the arithmetic, geometric, and Bayesian averages of the number of responses observed in FCN-10 and FCN-20 trials.

A flexible and generalizable model of online latent-state learning

Many models of classical conditioning fail to describe important phenomena, notably the rapid return of fear after extinction. To address this shortfall, evidence converged on the idea that learning agents rely on latent-state inferences, i.e. an ability to index disparate associations from cues to rewards (or penalties) and infer which index (i.e. latent state) is presently active. Our goal was to develop a model of latent-state inferences that uses latent states to predict rewards from cues efficiently and that can describe behavior in a diverse set of experiments. The resulting model combines a Rescorla-Wagner rule, for which updates to associations are proportional to prediction error, with an approximate Bayesian rule, for which beliefs in latent states are proportional to prior beliefs and an approximate likelihood based on current associations. In simulation, we demonstrate the model's ability to reproduce learning effects both famously explained and not explained by the Rescorla-Wagner model, including rapid return of fear after extinction, the Hall-Pearce effect, partial reinforcement extinction effect, backwards blocking, and memory modification. Lastly, we derive our model as an online algorithm to maximum likelihood estimation, demonstrating it is an efficient approach to outcome prediction. Establishing such a framework is a key step towards quantifying normative and pathological ranges of latent-state inferences in various contexts.

How competitive is cue competition?

Cue competition refers to phenomena indicating that learning about the relationship between a cue and an outcome is influenced by learning about the predictive significance of other cues that are concurrently present. In two autoshaping experiments with pigeons, we investigated the strength of competition among cues for predictive value. In each experiment, animals received an overexpectation training (A+, D+ followed by AD+). In addition, the training schedule of each experiment comprised two control conditions-one condition to evaluate the presence of overexpectation (B+ followed by BY+) and a second one to assess the strength of competition among cues (C+ followed by CZ-). Training trials were followed by a test with individual stimuli (A, B, C). Experiment 1 revealed no evidence for cue competition as responding during the test mirrored the individual cue-outcome contingencies. The test results from Experiment 2, which included an outcome additivity training, showed cue competition in form of an overexpectation effect as responding was weaker for Stimulus A than Stimulus B. However, the test results from Experiment 2 also revealed that responding to Stimulus A was stronger than to Stimulus C, which indicates that competition among cues was not as strong as predicted by some influential theories of associative learning.

The biological and behavioral computations that influence dopamine responses

Phasic dopamine responses demonstrate remarkable simplicity; they code for the differences between received and predicted reward values. Yet this simplicity belies the subtle complexity of the psychological, computational, and contextual factors that influence this signal. Advances in behavioral paradigms and models, in monkeys and rodents, have demonstrated that phasic dopamine responses reflect numerous behavioral computations and factors including choice, subjective value, confidence, and context. The application of optogenetics has provided evidence that dopamine reward prediction error responses cause value learning. Furthermore, studies using advanced circuit tracing techniques have begun to uncover the biological network implementation of the reward learning algorithm. The purpose of this review is to summarize the recent advances in dopamine neurophysiology and synthesize an updated account of the behavioral function of dopamine signals.

The Role of Associative History in Models of Associative Learning: A Selective Review and a Hybrid Model

Associative learning theories strive to capture the processes underlying and driving the change in strength of the associations between representations of stimuli that develop as a result of experience of the predictive relationships between those stimuli. Historically, formal models of associative learning have focused on two potential factors underlying associative change, namely processing of the conditioned stimulus (in terms of changes in associability) and processing of the unconditioned stimulus (in terms of changes in error). This review constitutes an analysis of the proper role of these two factors, specifically with regard to the way in which they are influenced by associative history (the prior training undergone by cues). A novel “hybrid” model of associative learning is proposed and is shown to provide a more satisfactory account of the effects of associative history on subsequent learning than any previous single-process theory.

Purkinje cell activity during classical conditioning with different conditional stimuli explains central tenet of Rescorla–Wagner model [corrected]

A central tenet of Rescorla and Wagner's model of associative learning is that the reinforcement value of a paired trial diminishes as the associative strength between the presented stimuli increases. Despite its fundamental importance to behavioral sciences, the neural mechanisms underlying the model have not been fully explored. Here, we present findings that, taken together, can explain why a stronger association leads to a reduced reinforcement value, within the context of eyeblink conditioning. Specifically, we show that learned pause responses in Purkinje cells, which trigger adaptively timed conditioned eyeblinks, suppress the unconditional stimulus (US) signal in a graded manner. Furthermore, by examining how Purkinje cells respond to two distinct conditional stimuli and to a compound stimulus, we provide evidence that could potentially help explain the somewhat counterintuitive overexpectation phenomenon, which was derived from the Rescorla-Wagner model.

Implications of learning theory for developing programs to decrease overeating

Childhood obesity is associated with medical and psychological comorbidities, and interventions targeting overeating could be pragmatic and have a significant impact on weight. Calorically dense foods are easily available, variable, and tasty which allows for effective opportunities to learn to associate behaviors and cues in the environment with food through fundamental conditioning processes, resulting in measurable psychological and physiological food cue reactivity in vulnerable children. Basic research suggests that initial learning is difficult to erase, and that it is vulnerable to a number of phenomena that will allow the original learning to re-emerge after it is suppressed or replaced. These processes may help explain why it may be difficult to change food cue reactivity and overeating over the long term. Extinction theory may be used to develop effective cue-exposure treatments to decrease food cue reactivity through inhibitory learning, although these processes are complex and require an integral understanding of the theory and individual differences. Additionally, learning theory can be used to develop other interventions that may prove to be useful. Through an integration of learning theory, basic and translational research, it may be possible to develop interventions that can decrease the urges to overeat, and improve the weight status of children.

Effects of prior cocaine versus morphine or heroin self-administration on extinction learning driven by overexpectation versus omission of reward

BACKGROUND

Addiction is characterized by an inability to stop using drugs, despite adverse consequences. One contributing factor to this compulsive drug taking could be the impact of drug use on the ability to extinguish drug seeking after changes in expected outcomes. Here, we compared effects of cocaine, morphine, and heroin self-administration on two forms of extinction learning: standard extinction driven by reward omission and extinction driven by reward overexpectation.

METHODS

In experiment 1, we trained rats to self-administer cocaine, morphine, or sucrose for 3 hours per day (limited access). In experiment 2, we trained rats to self-administer heroin or sucrose for 12 hours per day (extended access). Three weeks later, we trained the rats to associate several cues with palatable food reward, after which we assessed extinction of the learned Pavlovian response, first by pairing two cues together in the overexpectation procedure and later by omitting the food reward.

RESULTS

Rats trained under limited access conditions to self-administer sucrose or morphine demonstrated normal extinction in response to both overexpectation and reward omission, whereas cocaine-experienced rats or rats trained to self-administer heroin under extended access conditions exhibited normal extinction in response to reward omission but failed to show extinction in response to overexpectation.

CONCLUSIONS

Here we show that cocaine and heroin can induce long-lasting deficits in the ability to extinguish reward seeking. These deficits were not observed in a standard extinction procedure but instead only affected extinction learning driven by a more complex phenomenon of overexpectation.

Accounting for negative automaintenance in pigeons: a dual learning systems approach and factored representations

Animals, including Humans, are prone to develop persistent maladaptive and suboptimal behaviours. Some of these behaviours have been suggested to arise from interactions between brain systems of Pavlovian conditioning, the acquisition of responses to initially neutral stimuli previously paired with rewards, and instrumental conditioning, the acquisition of active behaviours leading to rewards. However the mechanics of these systems and their interactions are still unclear. While extensively studied independently, few models have been developed to account for these interactions. On some experiment, pigeons have been observed to display a maladaptive behaviour that some suggest to involve conflicts between Pavlovian and instrumental conditioning. In a procedure referred as negative automaintenance, a key light is paired with the subsequent delivery of food, however any peck towards the key light results in the omission of the reward. Studies showed that in such procedure some pigeons persisted in pecking to a substantial level despite its negative consequence, while others learned to refrain from pecking and maximized their cumulative rewards. Furthermore, the pigeons that were unable to refrain from pecking could nevertheless shift their pecks towards a harmless alternative key light. We confronted a computational model that combines dual-learning systems and factored representations, recently developed to account for sign-tracking and goal-tracking behaviours in rats, to these negative automaintenance experimental data. We show that it can explain the variability of the observed behaviours and the capacity of alternative key lights to distract pigeons from their detrimental behaviours. These results confirm the proposed model as an interesting tool to reproduce experiments that could involve interactions between Pavlovian and instrumental conditioning. The model allows us to draw predictions that may be experimentally verified, which could help further investigate the neural mechanisms underlying theses interactions.

Contemporary Study of Pavlovian Conditioning

Pavlov's first report on conditioning emphasized its role in allowing the animal to adjust to its environment. Contemporary theories have seen this adjustment in terms of developing accurate knowledge of the environment. Three aspects of that thinking are explored: how the animal acquires initial knowledge, how it changes its knowledge when conditions of the world change, and how it makes use of multiple knowledge representations.

Modelling individual differences in the form of Pavlovian conditioned approach responses: a dual learning systems approach with factored representations

Reinforcement Learning has greatly influenced models of conditioning, providing powerful explanations of acquired behaviour and underlying physiological observations. However, in recent autoshaping experiments in rats, variation in the form of Pavlovian conditioned responses (CRs) and associated dopamine activity, have questioned the classical hypothesis that phasic dopamine activity corresponds to a reward prediction error-like signal arising from a classical Model-Free system, necessary for Pavlovian conditioning. Over the course of Pavlovian conditioning using food as the unconditioned stimulus (US), some rats (sign-trackers) come to approach and engage the conditioned stimulus (CS) itself - a lever - more and more avidly, whereas other rats (goal-trackers) learn to approach the location of food delivery upon CS presentation. Importantly, although both sign-trackers and goal-trackers learn the CS-US association equally well, only in sign-trackers does phasic dopamine activity show classical reward prediction error-like bursts. Furthermore, neither the acquisition nor the expression of a goal-tracking CR is dopamine-dependent. Here we present a computational model that can account for such individual variations. We show that a combination of a Model-Based system and a revised Model-Free system can account for the development of distinct CRs in rats. Moreover, we show that revising a classical Model-Free system to individually process stimuli by using factored representations can explain why classical dopaminergic patterns may be observed for some rats and not for others depending on the CR they develop. In addition, the model can account for other behavioural and pharmacological results obtained using the same, or similar, autoshaping procedures. Finally, the model makes it possible to draw a set of experimental predictions that may be verified in a modified experimental protocol. We suggest that further investigation of factored representations in computational neuroscience studies may be useful.

Behavioral and neurobiological mechanisms of extinction in Pavlovian and instrumental learning

This article reviews research on the behavioral and neural mechanisms of extinction as it is represented in both Pavlovian and instrumental learning. In Pavlovian extinction, repeated presentation of a signal without its reinforcer weakens behavior evoked by the signal; in instrumental extinction, repeated occurrence of a voluntary action without its reinforcer weakens the strength of the action. In either case, contemporary research at both the behavioral and neural levels of analysis has been guided by a set of extinction principles that were first generated by research conducted at the behavioral level. The review discusses these principles and illustrates how they have informed the study of both Pavlovian and instrumental extinction. It shows that behavioral and neurobiological research efforts have been tightly linked and that their results are readily integrated. Pavlovian and instrumental extinction are also controlled by compatible behavioral and neural processes. Since many behavioral effects observed in extinction can be multiply determined, we suggest that the current close connection between behavioral-level and neural-level analyses will need to continue.

Over-expectation generated in a complex appetitive goal-tracking task is capable of inducing memory reconsolidation

RATIONALE

Discrepancies in an expected outcome have been demonstrated to result in modification of behaviour in both appetitive and aversive conditioning settings.

OBJECTIVES

In this study, we sought to establish whether overexpectation generated from compound conditioning with two previously rewarded stimuli was able to induce memory destabilisation and subsequent reconsolidation in a Pavlovian conditioned approach setting.

RESULTS

It was shown that 4 days, but not 1 day, of overexpectation training was required to induce memory reconsolidation, and this was disrupted by application of the NMDA subtype of glutamate receptor antagonist MK-801 prior to overexpectation training, but not by MK-801 application 6 h post-training.

CONCLUSIONS

These data provide evidence that the memories underlying Pavlovian conditioned approach do undergo reconsolidation and that such reconsolidation can be triggered by overexpectation. Therefore, the updating of appetitive conditioned stimulus and unconditioned stimulus associations underpinning conditioned responding in manners other than extinction training is likely achieved through memory reconsolidation.

Normalization between stimulus elements in a model of Pavlovian conditioning: showjumping on an elemental horse

Harris and Livesey. Learning & Behavior, 38, 1-26, (2010) described an elemental model of associative learning that implements a simple learning rule that produces results equivalent to those proposed by Rescorla and Wagner (1972), and additionally modifies in "real time" the strength of the associative connections between elements. The novel feature of this model is that stimulus elements interact by suppressively normalizing one another's activation. Because of the normalization process, element activity is a nonlinear function of sensory input strength, and the shape of the function changes depending on the number and saliences of all stimuli that are present. The model can solve a range of complex discriminations and account for related empirical findings that have been taken as evidence for configural learning processes. Here we evaluate the model's performance against the host of conditioning phenomena that are outlined in the companion article, and we present a freely available computer program for use by other researchers to simulate the model's behavior in a variety of conditioning paradigms.

Facets of Pavlovian and operant extinction

Research on extinction is of fundamental importance in both Pavlovian and operant approaches to the experimental analysis of learning. Although these approaches are often motivated by different empirical and theoretical questions, extinction has emerged as a research area in which common themes unite the two approaches. In this review, we focus on some common considerations in the analysis of Pavlovian and operant extinction. These include methodological challenges and interpretational issues in analyzing behavior during and after extinction. We consider the different roles that theory has played in the development of research on extinction in these preparations and conclude with some attention to applications of extinction.

A test of Rescorla and Wagner's (1972) prediction of nonlinear effects in contingency learning

According to most theories, in a simple contingency learning situation, excitatory learning occurs when the probability of the unconditioned stimulus in the presence of the conditioned stimulus (p(1)) is higher than the probability of the unconditioned stimulus in the absence of the conditioned stimulus (p(2)). In Rescorla and Wagner's (1972) model, this prediction varies, depending on the parameters used. In the following experiments, we evaluated whether the difference between p(1) and p(2) that is required to produce excitatory conditioning is the same, independent of the specific value of p(1), or whether this difference varies proportionally to p(1)'s value. To do so, an appetitive procedure of Pavlovian conditioning with rats was used. In four experiments, we compared different levels of contingency (low, medium and high) and found that the difference between p(1) and p(2) that is required to produce excitatory conditioning increases when the value of p(1) is higher. The possibility of analyzing contingency learning as a discrimination between p(1) and p(2) is also discussed.

Decoding different roles for vmPFC and dlPFC in multi-attribute decision making

In everyday life, successful decision making requires precise representations of expected values. However, for most behavioral options more than one attribute can be relevant in order to predict the expected reward. Thus, to make good or even optimal choices the reward predictions of multiple attributes need to be integrated into a combined expected value. Importantly, the individual attributes of such multi-attribute objects can agree or disagree in their reward prediction. Here we address where the brain encodes the combined reward prediction (averaged across attributes) and where it encodes the variability of the value predictions of the individual attributes. We acquired fMRI data while subjects performed a task in which they had to integrate reward predictions from multiple attributes into a combined value. Using time-resolved pattern recognition techniques (support vector regression) we find that (1) the combined value is encoded in distributed fMRI patterns in the ventromedial prefrontal cortex (vmPFC) and that (2) the variability of value predictions of the individual attributes is encoded in the dorsolateral prefrontal cortex (dlPFC). The combined value could be used to guide choices, whereas the variability of the value predictions of individual attributes indicates an ambiguity that results in an increased difficulty of the value-integration. These results demonstrate that the different features defining multi-attribute objects are encoded in non-overlapping brain regions and therefore suggest different roles for vmPFC and dlPFC in multi-attribute decision making.

The orbitofrontal cortex and ventral tegmental area are necessary for learning from unexpected outcomes

Humans and other animals change their behavior in response to unexpected outcomes. The orbitofrontal cortex (OFC) is implicated in such adaptive responding, based on evidence from reversal tasks. Yet these tasks confound using information about expected outcomes with learning when those expectations are violated. OFC is critical for the former function; here we show it is also critical for the latter. In a Pavlovian overexpectation task, inactivation of OFC prevented learning driven by unexpected outcomes, even when performance was assessed later. We propose this reflects a critical contribution of outcome signaling by OFC to encoding of reward prediction errors elsewhere. In accord with this proposal, we report that signaling of reward predictions by OFC neurons was related to signaling of prediction errors by dopamine neurons in ventral tegmental area (VTA). Furthermore, bilateral inactivation of VTA or contralateral inactivation of VTA and OFC disrupted learning driven by unexpected outcomes.

Renewal after overexpectation

Four magazine approach experiments were performed with rats to explore the decrement produced by reinforcing a compound of two previously conditioned stimuli. Compound presentation produced the usual over-expectation decrement in responding to the individual stimuli. However, testing in an alternative, but similarly treated, context allowed renewal of the initial responding to the individual stimuli. This renewal is similar to that seen after a decrement produced by nonreinforcement. It joins other results in suggesting that the decrement produced in extinction and overexpectation may be due to the same mechanism.

Summation in causal learning: elemental processing or configural generalization?

Results from human causal learning tasks that employ multiple cues are often interpreted in terms of the elemental theory of Rescorla and Wagner (1972). However, most results can also be successfully interpreted by the configural model proposed by Pearce (1987, 1994). One method of discriminating between these alternatives is through an investigation of summation and overexpectation. Indeed, demonstrations of these phenomena are fundamental to an elemental approach but are generally incompatible with an account that involves configural processing. Using a procedure in which the magnitude of the outcome varied, evidence for both summation and overexpectation was obtained in two experiments.

Deepened extinction from compound stimulus presentation

Three experiments with rats and 2 with pigeons explored the effect of presenting 2 extinguished excitatory stimuli in compound. Four learning situations were used: Pavlovian magazine approach, Pavlovian fear conditioning, and instrumental discriminative instrumental learning in rats, as well as Pavlovian sign tracking in pigeons. All 5 experiments confirmed D. Reberg's (1972) observation that even after extinction of the individual stimuli, presenting them in compound evoked substantial responding. Moreover, nonreinforcement of that compound deepened extinction of an element more substantially than did additional presentation of that element alone. Such compound exposure reduced spontaneous recovery, reduced reinstatement, and slowed subsequent reconditioning. The primary determinant seemed to be the enhanced associative strength rather than the enhanced conditioned responding that occurred during the nonreinforced compound.

Predicting danger: the nature, consequences, and neural mechanisms of predictive fear learning

The ability to detect and learn about the predictive relations existing between events in the world is essential for adaptive behavior. It allows us to use past events to predict the future and to adjust our behavior accordingly. Pavlovian fear conditioning allows anticipation of sources of danger in the environment. It guides attention away from poorer predictors toward better predictors of danger and elicits defensive behavior appropriate to these threats. This article reviews the differences between learning about predictive relations and learning about contiguous relations in Pavlovian fear conditioning. It then describes behavioral approaches to the study of these differences and to the examination of subtle variations in the nature and consequences of predictive learning. Finally, it reviews recent data from rodent and human studies that have begun to identify the neural mechanisms for direct and indirect predictive fear learning.

Spontaneous recovery from overexpectation

In three Pavlovian magazine approach experiments, rats received conditioning of auditory and visual stimuli by pairing with a pellet. Then the stimuli received additional conditioning while presented in simultaneous compound and were tested either immediately or after a delay. The compound conditioning resulted in a decrement in responding to the individual stimuli (overexpectation). However, there was recovery of responding with the passage of time. These results suggest that the decrements produced by an overexpectation procedure share some properties with those produced by nonreinforcement.

Elemental representations of stimuli in associative learning

This article reviews evidence and theories concerning the nature of stimulus representations in Pavlovian conditioning. It focuses on the elemental approach developed in stimulus sampling theory (R. C. Atkinson & W. K. Estes, 1963; R. R. Bush & F. Mosteller, 1951b) and extended by I. P. L. McLaren and N. J. Mackintosh (2000, 2002) and contrasts this with models that invoke notions of configural representations that uniquely code for different patterns of stimulus inputs (e.g., J. M. Pearce, 1987, 1994; R. A. Rescorla & A. R. Wagner, 1972; A. R. Wagner & S. E. Brandon, 2001). The article then presents a new elemental model that emphasizes interactions between stimulus elements. This model is shown to explain a range of behavioral findings, including those (e.g., negative patterning and biconditional discriminations) traditionally thought to be beyond the explanatory capabilities of elemental models. Moreover, the model offers a ready explanation for recent findings reported by R. A. Rescorla (2000, 2001, 2002b) concerning the way that stimuli with different conditioning histories acquire associative strength when conditioned in compound.

Context and behavioral processes in extinction

This article provides a selective review and integration of the behavioral literature on Pavlovian extinction. The first part reviews evidence that extinction does not destroy the original learning, but instead generates new learning that is especially context-dependent. The second part examines insights provided by research on several related behavioral phenomena (the interference paradigms, conditioned inhibition, and inhibition despite reinforcement). The final part examines four potential causes of extinction: the discrimination of a new reinforcement rate, generalization decrement, response inhibition, and violation of a reinforcer expectation. The data are consistent with behavioral models that emphasize the role of generalization decrement and expectation violation, but would be more so if those models were expanded to better accommodate the finding that extinction involves a context-modulated form of inhibitory learning.

Overexpectation: response loss during sustained stimulus compounding in the rabbit nictitating membrane preparation

Rabbits were given reinforced training of the nictitating membrane (NM) response using separate conditioned stimuli (CSs), which were a tone, light, and/or tactile vibration. Then, two CSs were compounded and given further pairings with the unconditioned stimulus (US). Evidence of both overexpectation and summation effects appeared. That is, responding to the individual CSs declined despite their continued pairing with the US on compound trials (overexpectation), and responding on the compound trials was greater than responding to the individual CSs (summation). The response loss appeared regardless of the testing regime, that is, whether the test presentations of the individual CSs were themselves reinforced (Experiment 2), not reinforced (Experiment 1), or deferred until the end of compound training (Experiment 2). The results are discussed with respect to the roles of excitatory versus inhibitory processes, elemental versus configural processes, and the possible roles of cerebellar and hippocampal pathways.

Blocking, Unblocking, and Overexpectation of Fear: A Role for Opioid Receptors in the Regulation of Pavlovian Association Formation

Injection of the opioid receptor antagonist naloxone facilitated acquisition of fear to contextual and auditory conditioned stimuli (CSs) in Experiments 1A and 1B. Experiment 2 showed that prior conditioning to a distinctive context blocked conditioning to an auditory CS. Blocking of CS fear was prevented by administrations of naloxone or increases in footshock intensity. Blocking of CS fear was facilitated by decreases in footshock intensity in a naloxone-reversible manner. Experiment 3 showed that compound conditioning of two CSs, each previously and separately paired with shock, produced overexpectation of fear that was reversed by naloxone. These results are consistent with a role for opioid receptors controlling Pavlovian association formation by regulating the discrepancy (lambda - SigmaV) described by R. A. Rescorla and A. R. Wagner (1972).

CONDITIONING WITH COMPOUND STIMULI IN DROSOPHILA MELANOGASTER IN THE FLIGHT SIMULATOR

Short-term memory in Drosophila melanogaster operant visual learning in the flight simulator is explored using patterns and colours as a compound stimulus. Presented together during training, the two stimuli accrue the same associative strength whether or not a prior training phase rendered one of the two stimuli a stronger predictor for the reinforcer than the other(no blocking). This result adds Drosophila to the list of other invertebrates that do not exhibit the robust vertebrate blocking phenomenon. Other forms of higher-order learning, however, were detected: a solid sensory preconditioning and a small second-order conditioning effect imply that associations between the two stimuli can be formed, even if the compound is not reinforced.

Associative changes with a random CS-US relationship

Three experiments examined conditioned magazine approach in rats when a positive unconditioned stimulus (US) bore a random relation to a conditioned stimulus (CS). Experiment 1 found that over the course of conditioning the CS initially elevated responding relative to the baseline but then lost the power to do so. Transfer tests revealed that a CS-US association developed early and persisted despite the decline in magazine responding. Experiment 2 confirmed the persistence of CS-US associations and found them to be more substantial when a different US occurred during the CS than in its absence. In Experiment 3, when the situation was exposed to US-alone presentations prior to introducing the CS, there was little evidence that a subsequent random relation between the CS and US produced an association between them. These results agree with those of blocking and overshadowing experiments using discrete CSs and support an interpretation of the random procedure in terms of competition between the background and CS for conditioning.