Contextual conditioning during free-operant extinction: Unsignaled, signaled, and backward-signaled noncontingent food

A hippocampo-cortical pathway detects changes in the validity of an action as a predictor of reward

Much research has been dedicated to understanding the psychological and neural bases of goal-directed action, yet the relationship between context and goal-directed action is not well understood. Here, we used excitotoxic lesions, chemogenetics, and circuit-specific manipulations to demonstrate the role of the ventral hippocampus (vHPC) in contextual learning that supports sensitivity to action-outcome contingencies, a hallmark of goal-directed action. We found that chemogenetic inhibition of the ventral, but not dorsal, hippocampus attenuated sensitivity to instrumental contingency degradation. We then tested the hypothesis that this deficit was due to an inability to discern the relative validity of the action compared with the context as a predictor of reward. Using latent inhibition and Pavlovian context conditioning, we confirm that degradation of action-outcome contingencies relies on intact context-outcome learning and show that this learning is dependent on vHPC. Finally, we show that chemogenetic inhibition of vHPC terminals in the medial prefrontal cortex also impairs both instrumental contingency degradation and context-outcome learning. These results implicate a hippocampo-cortical pathway in adapting to changes in instrumental contingencies and indicate that the psychological basis of this deficit is an inability to learn the predictive value of the context. Our findings contribute to a broader understanding of the neural bases of goal-directed action and its contextual regulation.

Context, attention, and the switch between habit and goal-direction in behavior

This article reviews recent findings from the author’s laboratory that may provide new insights into how habits are made and broken. Habits are extensively practiced behaviors that are automatically evoked by antecedent cues and performed without their goal (or reinforcer) “in mind.” Goal-directed actions, in contrast, are instrumental behaviors that are performed because their goal is remembered and valued. New results suggest that actions may transition to habit after extended practice when conditions encourage reduced attention to the behavior. Consistent with theories of attention and learning, a behavior may command less attention (and become habitual) as its reinforcer becomes well-predicted by cues in the environment; habit learning is prevented if presentation of the reinforcer is uncertain. Other results suggest that habits are not permanent, and that goal-direction can be restored by several environmental manipulations, including exposure to unexpected reinforcers or context change. Habits are more context-dependent than goal-directed actions are. Habit learning causes retroactive interference in a way that is reminiscent of extinction: It inhibits, but does not erase, goal-direction in a context-dependent way. The findings have implications for the understanding of habitual and goal-directed control of behavior as well as disordered behaviors like addictions.

BEHAVIORAL AND NEUROBIOLOGICAL MECHANISMS OF PAVLOVIAN AND INSTRUMENTAL EXTINCTION LEARNING

This article reviews the behavioral neuroscience of extinction, the phenomenon in which a behavior that has been acquired through Pavlovian or instrumental (operant) learning decreases in strength when the outcome that reinforced it is removed. Behavioral research indicates that neither Pavlovian nor operant extinction depends substantially on erasure of the original learning but instead depends on new inhibitory learning that is primarily expressed in the context in which it is learned, as exemplified by the renewal effect. Although the nature of the inhibition may differ in Pavlovian and operant extinction, in either case the decline in responding may depend on both generalization decrement and the correction of prediction error. At the neural level, Pavlovian extinction requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in extinction retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. Instrumental extinction involves distinct ensembles in corticostriatal, striatopallidal, and striatohypothalamic circuits as well as their thalamic returns for inhibitory (extinction) and excitatory (renewal and other relapse phenomena) control over operant responding. The field has made significant progress in recent decades, although a fully integrated biobehavioral understanding still awaits.

Relapse of operant behavior after response elimination with an extinction or an omission contingency

The present study compared relapse after responding was eliminated by extinction or omission training in rats. In Experiment 1, lever pressing was reinforced with food pellets in Context A and then eliminated with either extinction or omission training in Context B. The response was then tested in Contexts A and B in either the presence or absence of free food pellets delivered on a random time schedule. All rats showed higher responding when tested in Context A than Context B, and there was little evidence that omission training attenuated this ABA renewal effect. Noncontingent pellets increased responding after extinction but not after omission. However, when responding on the last day of response elimination was compared to responding during the test in the response-elimination context, there was some evidence that omission-trained rats showed a small increase in responding even when tested with free pellets. Results of Experiment 2 suggest this increase was not due to differences in the temporal distribution of pellets during elimination and the test, and that the result might be due to mere removal of the omission contingency, but any such effect is small and difficult to detect statistically. The results provide new information about factors generating relapse after omission training.

Extinction of instrumental (operant) learning: interference, varieties of context, and mechanisms of contextual control

This article reviews recent research on the extinction of instrumental (or operant) conditioning from the perspective that it is an example of a general retroactive interference process. Previous discussions of interference have focused primarily on findings from Pavlovian conditioning. The present review shows that extinction in instrumental learning has much in common with other examples of retroactive interference in instrumental learning (e.g., omission learning, punishment, second-outcome learning, discrimination reversal learning, and differential reinforcement of alternative behavior). In each, the original learning can be largely retained after conflicting information is learned, and behavior is cued or controlled by the current context. The review also suggests that a variety of stimuli can play the role of context, including room and apparatus cues, temporal cues, drug state, deprivation state, stress state, and recent reinforcers, discrete cues, or behaviors. In instrumental learning situations, the context can control behavior through its direct association with the reinforcer or punisher, through its hierarchical relation with response-outcome associations, or its direct association (inhibitory or excitatory) with the response. In simple instrumental extinction and habit learning, the latter mechanism may play an especially important role.

Discriminative properties of the reinforcer can be used to attenuate the renewal of extinguished operant behavior

Previous research on the resurgence effect has suggested that reinforcers that are presented during the extinction of an operant behavior can control inhibition of the response. To further test this hypothesis, in three experiments with rat subjects we examined the effectiveness of using reinforcers that were presented during extinction as a means of attenuating or inhibiting the operant renewal effect. In Experiment 1, lever pressing was reinforced in Context A, extinguished in Context B, and then tested in Context A. Renewal of responding that occurred during the final test was attenuated when a distinct reinforcer that had been presented independent of responding during extinction was also presented during the renewal test. Experiment 2 established that this effect depended on the reinforcer being featured as a part of extinction (and thus associated with response inhibition). Experiment 3 then showed that the reinforcers presented during extinction suppressed performance in both the extinction and renewal contexts; the effects of the physical and reinforcer contexts were additive. Together, the results further suggest that reinforcers associated with response inhibition can serve a discriminative role in suppressing behavior and may be an effective stimulus that can attenuate operant relapse.

Role of the discriminative properties of the reinforcer in resurgence

In three experiments with rat subjects, we examined the effects of the discriminative effects of reinforcers that were presented during or after operant extinction. Experiments 1 and 2 examined resurgence, in which an extinguished operant response (R1) recovers when a second behavior (R2) that has been reinforced to replace it is also placed in extinction. The results of Experiment 1 suggest that the amount of R1's resurgence is a decreasing linear function of the interreinforcement interval used during the reinforcement of R2. In Experiment 2, R1 was reinforced with one outcome (O1), and R2 was then reinforced with a second outcome (O2) while R1 was extinguished. In resurgence tests, response-independent (noncontingent) presentations of O2 prevented resurgence of R1, which otherwise occurred when testing was conducted with either no reinforcers or noncontingent presentations of O1. In Experiment 3, we then examined the effects of noncontingent O1 and O2 presentations after simple extinction in either the presence or the absence of noncontingent presentations of O2. Overall, the results are consistent with a role for the discriminative properties of the reinforcer in controlling operant behavior. In resurgence, the reinforcer used during response elimination provides a distinct context that controls the inhibition of R1. The results are less consistent with an alternative view emphasizing the disrupting effects of alternative reinforcement.

Reinstatement of instrumental actions in humans: Possible mechanisms and their implications to prevent it

The study of post-extinction recovery effects in humans has received significant attention. For instance, research on reinstatement has increased in the last decade. However, most of the studies focus on the return of fear responses. In the present experiments, we used a videogame task to explore the reinstatement of operant behavior in human participants. In Experiment 1, after participants learned to shoot at enemies, they received an extinction procedure that eliminated the shooting behavior. However, the mere reintroduction of the outcome reinstated the original response. Experiment 2 showed that the reinstatement of instrumental behavior is contextually modulated. Finally, in Experiment 3 we found that presenting a reminder for extinction attenuated the response recovery effect. The overall pattern of results suggests that reinstatement of voluntary actions in humans could be explained by an interference memory framework. In addition, the present data suggest that therapies that use brief reminders of therapeutic intervention could help prevent the reinstatement of unhealthy instrumental behaviors.

Factors that influence the persistence and relapse of discriminated behavior chains

Behavior chains are composed of sequences of behaviors that minimally include procurement and then consumption. This review surveys recent research from this laboratory that has examined the properties of discriminated heterogeneous behavior chains. In contrast to another review (Thrailkill and Bouton, 2016a), it discusses work examining what makes chained behavior persistent, and what makes it relapse. Results suggest that responses in a discriminated heterogeneous behavior chain may become associated, so that extinction of either one reduces the strength of the other. Evidence also suggests that the goal of the first (procurement) response may be the next (consumption) response (rather than the upcoming discriminative stimulus, a putative conditioned reinforcer, or the primary reinforcer at the end of the chain). Further studies suggest that methods that promote generalization across acquisition and extinction (partial reinforcement and delivery of noncontingent reinforcers during extinction) lead to greater persistence of the procurement response. A third set of studies analyzed the contextual control and relapse of chained behaviors. The context controls both the acquisition and extinction of chained behaviors. In addition, a separately-extinguished consumption response is renewed when returned to the context of the chain. The research expands our general understanding of the learning processes that govern instrumental behavior as well as our understanding of chains.

Increasing the persistence of a heterogeneous behavior chain: Studies of extinction in a rat model of search behavior of working dogs

Dogs trained to search for contraband perform a chain of behavior in which they first search for a target and then make a separate response that indicates to the trainer that they have found one. The dogs often conduct multiple searches without encountering a target and receiving the reinforcer (i.e., no contraband is present). Understanding extinction (i.e., the decline in work rate when reinforcers are no longer encountered) may assist in training dogs to work in conditions where targets are rare. We therefore trained rats on a search-target behavior chain modeled on the search behavior of working dogs. A discriminative stimulus signaled that a search response (e.g., chain pull) led to a second stimulus that set the occasion for a target response (e.g., lever press) that was reinforced by a food pellet. In Experiment 1 training with longer search durations and intermittent (partial) reinforcement of searching (i.e. some trials had no target present) both led to more persistent search responding in extinction. The loss of search behavior in extinction was primarily dependent on the number of non-reinforced searches rather than time searching without reinforcement. In Experiments 2 and 3, delivery of non-contingent reinforcers during extinction increased search persistence provided they had also been presented during training. Thus, results with rats suggest that the persistence of working dog performance (or chained behavior generally) may be improved by training with partial reinforcement of searching and non-contingent reinforcement during both training and work (extinction).

Attenuation and reacquisition of foraging behavior in a patchy environment

The present experiment examined the attenuation and reacquisition of foraging behavior of pigeons on a restricted diet. Using the procedure of Roberts (1988), thirty-two feeders were arranged in four circular patches of eight feeders each. The feeders were baited so that each patch contained a different density of food. Pigeons learned to forage among the patches. During this initial acquisition phase, visits to the patches by the pigeons differed as a function of the food density of the patches. After this initial foraging training phase, each pigeon received one of two response elimination procedures. For half of the subjects, food was not present during the response elimination phase. For the remaining subjects, food was placed exclusively outside the patches during this phase. Foraging behavior decreased quickly and somewhat similarly for both conditions. During a final phase in which the retraining of the original foraging behavior occurred, the group given food outside the patches during response elimination provided evidence of superior foraging with respect to sensitivity to the different patch food densities. The results are discussed with regard to previously published response elimination effects.

Behavioral and neurobiological mechanisms of extinction in Pavlovian and instrumental learning

This article reviews research on the behavioral and neural mechanisms of extinction as it is represented in both Pavlovian and instrumental learning. In Pavlovian extinction, repeated presentation of a signal without its reinforcer weakens behavior evoked by the signal; in instrumental extinction, repeated occurrence of a voluntary action without its reinforcer weakens the strength of the action. In either case, contemporary research at both the behavioral and neural levels of analysis has been guided by a set of extinction principles that were first generated by research conducted at the behavioral level. The review discusses these principles and illustrates how they have informed the study of both Pavlovian and instrumental extinction. It shows that behavioral and neurobiological research efforts have been tightly linked and that their results are readily integrated. Pavlovian and instrumental extinction are also controlled by compatible behavioral and neural processes. Since many behavioral effects observed in extinction can be multiply determined, we suggest that the current close connection between behavioral-level and neural-level analyses will need to continue.

Facets of Pavlovian and operant extinction

Research on extinction is of fundamental importance in both Pavlovian and operant approaches to the experimental analysis of learning. Although these approaches are often motivated by different empirical and theoretical questions, extinction has emerged as a research area in which common themes unite the two approaches. In this review, we focus on some common considerations in the analysis of Pavlovian and operant extinction. These include methodological challenges and interpretational issues in analyzing behavior during and after extinction. We consider the different roles that theory has played in the development of research on extinction in these preparations and conclude with some attention to applications of extinction.

The role of contextual conditioning in the effect of reinforcer devaluation on instrumental performance by rats

Different groups of rats received different amounts of training to lever press for a food reinforcer before an aversion was conditioned to the food. This devaluation of the reinforcer reduced responding in both subsequent extinction and reinforced tests of responding to a degree that was independent of the amount of instrumental training. Moreover, interpolating context extinction between aversion conditioning and the extinction test reduced the magnitude of the devaluation effect, thereby indicating that Pavlovian contextual conditioning may play a role in the instrumental devaluation effect.

Occasional Reinforced Responses During Extinction Can Slow the Rate of Reacquisition of an Operant Response

Three experiments with rats examined reacquisition of an operant response after either extinction or a response-elimination procedure that included occasional reinforced responses during extinction. In each experiment, reacquisition was slower when response elimination had included occasional reinforced responses, although the effect was especially evident when responding was examined immediately following each response-reinforcer pairing during reacquisition (Experiments 2 and 3). An extinction procedure with added noncontingent reinforcers also slowed reacquisition (Experiment 3). The results are consistent with research in classical conditioning (Bouton, Woods, & Pineño, 2004) and suggest that rapid reacquisition after extinction is analogous to a renewal effect that occurs when reinforced responses signal a return to the conditioning context. Clinical implications are also discussed.

Common processes may contribute to extinction and habituation

Psychologists routinely attribute the characteristics of conditioned behavior to complicated cognitive processes. For example, many of the characteristics of behavior undergoing extinction have been attributed to retrieval from memory. The authors argue that these characteristics may result from the simpler process of habituation. In particular, conditioned responding may decrease during extinction partially because habituation occurs to the stimuli that control responding when those stimuli are presented repeatedly or for a prolonged time (e.g., the experimental context, the conditioned stimulus in classical conditioning). This idea is parsimonious, has face validity, and evokes only processes that are well established by other evidence. In addition, behavior undergoing extinction shows 12 of the fundamental properties of behavior undergoing habituation. However, this model probably cannot provide a complete theory of extinction. It provides no obvious explanation for some of the other characteristics of extinguished behavior.

Developing a technology for the use of operant extinction in clinical settings: an examination of basic and applied research

Extinction of operant behavior, which involves terminating the reinforcement contingency that maintains a response, is important to the development, generalization, and reduction of behavior in clinical settings. We review basic and applied research findings on variables that influence the direct and indirect effects of extinction and discuss the potential value of a general technology for the use of extinction. We suggest that current research findings are not sufficient for the development of a comprehensive, applied technology of extinction and provide extensive guidelines for further studies on factors that may affect the course of extinction in clinical settings.