Contextual control of discriminated operant behavior

Habit and persistence

Voluntary behaviors (operants) can come in two varieties: Goal-directed actions, which are emitted based on the remembered value of the reinforcer, and habits, which are evoked by antecedent cues and performed without the reinforcer's value in active memory. The two are perhaps most clearly distinguished with the reinforcer-devaluation test: Goal-directed actions are suppressed when the reinforcer is separately devalued and responding is tested in extinction, and habitual behaviors are not. But what is the function of habit learning? Habits are often thought to be strong and unusually persistent. The present selective review examines this idea by asking whether habits identified by the reinforcer-devaluation test are more resistant to extinction, resistant to the effects of other contingency change, vulnerable to relapse, resistant to the weakening effects of context change, or permanently in place once they are learned. Surprisingly little evidence supports the idea that habits are permanent or more persistent. Habits are more context-specific than goal-directed actions are. Methods that make behavior persistent do not necessarily work by encouraging habit. The function of habit learning may not be to make a behavior strong or more persistent but to make it automatic and efficient in a particular context.

Contribution of dorsal versus ventral hippocampus to the hierarchical modulation of goal-directed actions in rats

Adaptive behaviour often necessitates that animals learn about events in a manner that is specific to a particular context or environment. These hierarchical organisations allow the animal to decide which action is the most appropriate when faced with ambiguous or conflicting possibilities. This study examined the role of hippocampus in enabling animals to use the context to guide action selection. We used a hierarchical instrumental outcome devaluation task in which male rats learn that the context provides information about the unique action-outcome relations that are in effect. We first confirmed that rats encode and use hierarchical context-(action-outcome) relations. We then show that chemogenetic inhibition of ventral hippocampus impairs both the encoding and retrieval of these associations, while inhibition of dorsal hippocampus impairs only the retrieval. Importantly, neither dorsal nor ventral hippocampus was required for goal-directed behaviour per se as these impairments only emerged when rats were forced to use the context to identify the current action-outcome relationships. These findings are discussed with respect to the role of the hippocampus and its broader circuitry in the contextual modulation of goal-directed behaviour and the importance of hierarchical associations in flexible behaviour.

Partial reinforcement extinction and omission effects in the elimination and recovery of discriminated operant behavior

Three experiments explored how training reinforcement schedules and context influence the elimination and recovery of human operant behavior. In Experiment 1, participants learned a discriminated operant response in Context A before the response was eliminated with extinction in Context B. They then received a final test in each context. Groups were trained with a discriminative stimulus that predicted a reinforced response on either every trial (continuous reinforcement [CRF]) or some of the trials (partial reinforcement [PRF]). Extinction was slower following PRF training (a partial reinforcement extinction effect [PREE]) and extinguished responding increased when tested in Context A ("ABA" renewal). Experiment 2 further confirmed the PREE was obtained equally whether extinction occurred in the training context (Context A) or a new context (Context B) which is consistent with trial-based accounts of the PREE. Experiment 3 used the same design as Experiment 1 to evaluate the influence of training reinforcement on response elimination with an omission contingency. Across the omission training phase in Context B, the decrease in responding occurred more slowly in the PRF-trained group in comparison to the CRF-trained group, perhaps the first demonstration of what might be termed a PRF omission effect. Again, ABA renewal was observed in Context A. Training reinforcement schedule therefore had a similar influence on response elimination with extinction and omission. Elimination and recovery of human instrumental behavior, with extinction or omission, are influenced by training reinforcement schedule and context. (PsycInfo Database Record (c) 2023 APA, all rights reserved).

Prelimbic cortex inactivation prevents ABA renewal based on satiety state

We have previously shown that the rat prelimbic cortex (PL) is necessary for contexts to promote the performance of instrumental behaviors that have been learned in them, whether the context is physical (operant chamber) or behavioral (recent performance of a behavior that has historically preceded the target in a behavior chain). In the present experiment, we investigated the role of the PL in satiety level as an interoceptive acquisition context. Rats were trained to lever-press for sweet/fat pellets while sated (22 hrs continuous food access) followed by the extinction of the response while hungry (22 hrs food deprived). Pharmacological inactivation of the PL (with baclofen/muscimol infusion) attenuated renewal of the response that occurred upon a return to the sated context. In contrast, animals that received a vehicle (saline) infusion showed renewal of the previously extinguished response. These results support the hypothesis that the PL monitors the relevant contextual elements (physical, behavioral, or satiety state) associated with reinforcement of a response and promotes the subsequent performance of that response in their presence.

Explaining the Return of Fear with Revised Rescorla-Wagner Models

Exposure therapy - exposure to a feared stimulus without harmful consequences - can reduce fear responses in many mental disorders. However, such relief is often partial and temporary: fear can return after the therapy has ended. Conditioning research has identified three mechanisms for the return of fear, viz. change in physical context (renewal), the passage of time (spontaneous recovery), and an encounter with the fear-producing unconditioned stimulus (reinstatement). To understand why fear returns and thereby develop more effective therapies, we develop mathematical learning models based on that of Rescorla and Wagner. According to this model, context cues present during extinction become conditioned inhibitors (i.e. safety signals) which prevent total erasure of the threat association. Adding various mechanisms to the model allows it to explain different facets of the return of fear. Among these mechanisms is decay of inhibitory associations, which provides a novel explanation for spontaneous recovery. To make the benefits of exposure robust and permanent, one must minimize the degree to which the extinction context becomes inhibitory in order to maximize unlearning. We simulate several experimental paradigms that reduce the return of fear and explain them according to this principle.

Context, attention, and the switch between habit and goal-direction in behavior

This article reviews recent findings from the author’s laboratory that may provide new insights into how habits are made and broken. Habits are extensively practiced behaviors that are automatically evoked by antecedent cues and performed without their goal (or reinforcer) “in mind.” Goal-directed actions, in contrast, are instrumental behaviors that are performed because their goal is remembered and valued. New results suggest that actions may transition to habit after extended practice when conditions encourage reduced attention to the behavior. Consistent with theories of attention and learning, a behavior may command less attention (and become habitual) as its reinforcer becomes well-predicted by cues in the environment; habit learning is prevented if presentation of the reinforcer is uncertain. Other results suggest that habits are not permanent, and that goal-direction can be restored by several environmental manipulations, including exposure to unexpected reinforcers or context change. Habits are more context-dependent than goal-directed actions are. Habit learning causes retroactive interference in a way that is reminiscent of extinction: It inhibits, but does not erase, goal-direction in a context-dependent way. The findings have implications for the understanding of habitual and goal-directed control of behavior as well as disordered behaviors like addictions.

Effect of context on the instrumental reinforcer devaluation effect produced by taste-aversion learning

Four experiments manipulated the context in which taste-aversion conditioning occurred when the reinforcer was devalued after instrumental learning. In all experiments, rats learned to lever press in an operant conditioning chamber and then had an aversion to the food-pellet reinforcer conditioned by pairing it with lithium chloride (LiCl) in either that context or a different context. Lever pressing was then tested in extinction to assess its status as a goal-directed action. In Experiment 1, aversion conditioning in the operant conditioning chamber suppressed lever-pressing during the test, but aversion conditioning in the home cage did not. Exposure to the averted pellet in the operant conditioning chamber after conditioning in the home cage did not change this effect (Experiment 2). The same pattern was observed when the different context was a second operant-style chamber (counterbalanced), exposure to the contexts was controlled, and pellets were presented in them in the same manner (Experiment 3). The greater effect of aversion conditioning in the instrumental context was not merely due to potentiated contextual conditioning (Experiment 4). Importantly, consumption tests revealed that the aversion conditioned in the different context had transferred to the test context. Thus, when reinforcer devaluation occurred in a different context, the rats lever pressed in extinction for a reinforcer they would otherwise reject. The results suggest that animals encode contextual information about the reinforcer during instrumental learning and suggest caution in making inferences about action versus habit learning when the reinforcer is devalued in a different context. (PsycInfo Database Record (c) 2021 APA, all rights reserved).

When Do We Not Face Our Fears? Investigating the Boundary Conditions of Costly Pain-Related Avoidance Generalization

Excessive generalization of fear and avoidance are hallmark symptoms of chronic pain disability, yet research focusing on the mechanisms underlying generalization of avoidance specifically, is scarce. Two experiments investigated the boundary conditions of costly pain-related avoidance generalization in healthy participants who learned to avoid pain by performing increasingly effortful (in terms of deviation and force) arm-movements using a robot-arm (acquisition). During generalization, novel, but similar arm-movements, without pain, were tested. Experiment 1 (N = 64) aimed to facilitate generalization to these movements by reducing visual contextual changes between acquisition and generalization, whereas Experiment 2 (N = 70) aimed to prevent extinction by increasing pain uncertainty. Both experiments showed generalization of pain-expectancies and pain-related fear. However, Experiment 2 was the first and only to also demonstrate generalization of avoidance, ie, choosing the novel effortful arm-movements in the absence of pain. These results suggest that uncertainty about the occurrence of pain may delay recovery, due to reduced disconfirmation of threat beliefs when exploring, resulting in persistent avoidance. PERSPECTIVE: This article demonstrates generalization of instrumentally acquired costly pain-related avoidance in healthy people under conditions of uncertainty. The results suggest that targeting pain-related uncertainty may be a useful tool for clinicians adopting a psychological approach to treating excessive pain-related avoidance in chronic pain.

Evaluating Extinction, Renewal, and Resurgence of Operant Behavior in Humans with Amazon Mechanical Turk

Amazon Mechanical Turk (MTurk) is a crowdsourcing marketplace providing researchers with the opportunity to collect behavioral data from remote participants at a low cost. Recent research demonstrated reliable extinction effects, as well as renewal and resurgence of button pressing with MTurk participants. To further examine the generality of these findings, we replicated and extended these methods across six experiments arranging reinforcement and extinction of a target button press. In contrast to previous findings, we did not observe as reliable of decreases in button pressing during extinction (1) after training with VR or VI schedules of reinforcement, (2) in the presence or absence of context changes, or (3) with an added response cost for button pressing. However, we found that that a 1-point response cost for all button presses facilitated extinction to a greater extent than the absence of response cost. Nevertheless, we observed ABA renewal of button pressing when changing background contexts across phases and resurgence when extinguishing presses on an alternative button. Our findings suggest that MTurk could be a viable platform from which to ask and address questions about extinction and relapse processes, but further procedural refinements will be necessary to improve the replicability of control by experimental contingencies.

BEHAVIORAL AND NEUROBIOLOGICAL MECHANISMS OF PAVLOVIAN AND INSTRUMENTAL EXTINCTION LEARNING

This article reviews the behavioral neuroscience of extinction, the phenomenon in which a behavior that has been acquired through Pavlovian or instrumental (operant) learning decreases in strength when the outcome that reinforced it is removed. Behavioral research indicates that neither Pavlovian nor operant extinction depends substantially on erasure of the original learning but instead depends on new inhibitory learning that is primarily expressed in the context in which it is learned, as exemplified by the renewal effect. Although the nature of the inhibition may differ in Pavlovian and operant extinction, in either case the decline in responding may depend on both generalization decrement and the correction of prediction error. At the neural level, Pavlovian extinction requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in extinction retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. Instrumental extinction involves distinct ensembles in corticostriatal, striatopallidal, and striatohypothalamic circuits as well as their thalamic returns for inhibitory (extinction) and excitatory (renewal and other relapse phenomena) control over operant responding. The field has made significant progress in recent decades, although a fully integrated biobehavioral understanding still awaits.

Repeated resurgence with and without a context change

Resurgence is the reoccurrence of a target response when reinforcement for a more recently reinforced alternative response is eliminated or reduced. The present study arranged two successive three-phase procedures to assess whether resurgence decreases with repeated assessments. Moreover, we arranged a contextual change from the first to second assessment for some groups. Phase 1 reinforced a target response on a touchscreen computer with typically developing adults as participants according to either variable-ratio or variable-interval schedules of reinforcement. Phase 2 extinguished target responding and reinforced alternative responding. Phase 3 tested for resurgence by extinguishing alternative responding. Resurgence reliably occurred in all tests and decreased from the first to second exposure to the procedures but there were no effects of context change. Therefore, repeated exposures to resurgence tests reduced those effects but contextual changes had no effect.

The impacts of acquisition and extinction cues on ABC renewal of voluntary behaviors

In two instrumental conditioning experiments with rats, we examined the impacts of acquisition and extinction cues on ABC renewal of instrumental behavior. Animals were reinforced with food for lever pressing in one context, followed by extinction of the response in a second one. Presentations of a brief tone accompanied extinction in Experiment 1 (extinction cue), and acquisition in Experiment 2 (acquisition cue). A final test in a third context revealed that instrumental responding was decreased in the presence of the extinction cue, whereas it was increased in the presence of the acquisition cue. We discuss theoretical and clinical implications of our results.

Discriminative control of saccade latencies

Recent studies have demonstrated that saccadic reaction times (SRTs) are influenced by the temporal regularities of dynamic environments (Vullings & Madelain, 2018). Here, we ask whether discriminative control (i.e., the possibility to use external stimuli signaling the future state of the environment) of latencies in a search task might be established using reinforcement contingencies. Eight participants made saccades within 80-750 ms toward a target displayed among distractors. We constructed two latency classes, "short" and "long," using the first and last quartiles of the individual baseline distributions. We then used a latency-contingent display paradigm in which finding the visual target among other items was made contingent upon specific SRTs. For a first group, the postsaccadic target was displayed only following short latencies with leftward saccades, and following long latencies with rightward saccades. The opposite was true for a second group. When short- and long-latency saccades were reinforced (i.e., the target was displayed) depending on the saccade direction, median latencies differed by 74 ms on average (all outside the 98% null hypothesis confidence intervals). Posttraining, in the absence of reinforcement, we still observed strong differences in latency distributions, averaging 64 ms for leftward versus rightward saccades. Our results demonstrate the discriminative control of SRTs, further supporting the effects of reinforcement learning for saccade. This study reveals that saccade triggering is finely controlled by learned temporal and spatial properties of the environment using predictive mechanisms.

Retrieval practice after multiple context changes, but not long retention intervals, reduces the impact of a final context change on instrumental behavior

Recent evidence from this laboratory suggests that a context switch after operant learning consistently results in a decrement in responding. One way to reduce this decrement is to train the response in multiple contexts. One interpretation of this result, rooted in stimulus sampling theory, is that conditioning of a greater number of common stimulus elements arising from more contexts causes better generalization to new contexts. An alternative explanation is that each change of context causes more effortful retrieval, and practice involving effortful retrieval results in learning that is better able to transfer to new situations. The current experiments were designed to differentiate between these two explanations for the first time in an animal learning and memory task. Experiment 1 demonstrated that the detrimental impact of a context change on an instrumental nose-poking response can be reduced by training the response in multiple contexts. Experiment 2 then found that a training procedure which inserted extended retention intervals between successive training sessions did not reduce the detrimental impact of a final context change. This occurred even though the inserted retention intervals had a detrimental impact on responding (and, thus, presumably retrieval) similar to the effect that context switches had in Experiment 1. Together, the results suggest that effortful retrieval practice may not be sufficient to reduce the negative impact of a context change on instrumental behavior. A common elements explanation which supposes that physical and temporal contextual cues do not overlap may account for the findings more readily.

Extinction of instrumental (operant) learning: interference, varieties of context, and mechanisms of contextual control

This article reviews recent research on the extinction of instrumental (or operant) conditioning from the perspective that it is an example of a general retroactive interference process. Previous discussions of interference have focused primarily on findings from Pavlovian conditioning. The present review shows that extinction in instrumental learning has much in common with other examples of retroactive interference in instrumental learning (e.g., omission learning, punishment, second-outcome learning, discrimination reversal learning, and differential reinforcement of alternative behavior). In each, the original learning can be largely retained after conflicting information is learned, and behavior is cued or controlled by the current context. The review also suggests that a variety of stimuli can play the role of context, including room and apparatus cues, temporal cues, drug state, deprivation state, stress state, and recent reinforcers, discrete cues, or behaviors. In instrumental learning situations, the context can control behavior through its direct association with the reinforcer or punisher, through its hierarchical relation with response-outcome associations, or its direct association (inhibitory or excitatory) with the response. In simple instrumental extinction and habit learning, the latter mechanism may play an especially important role.

Can the discriminative stimulus effects of nicotine function concurrently as modulatory opponents in operant and pavlovian occasion setting paradigms in rats?

Nicotine promotes interoceptive changes in the nervous system. Such interoceptive stimuli play important roles in modulating addictive behavior. Operant and Pavlovian stimulus control modulate responsiveness to environmental stimuli related to drug-seeking and self-administration. Nicotine functions as a discriminative stimulus in modulating operant behavior as well as Pavlovian feature stimuli in modulating the conditional responding (CR) to exteroceptive CS→US contingencies. Elucidation of the interaction of these interoceptive stimulus control functions is vital for a comprehensive understanding of nicotine use/abuse, which might lead to better behavioral treatment strategies. This experiment evaluated the interaction among Pavlovian feature positive (FP) and feature negative (FN) effects of nicotine on concurrently occurring operant S^D and S^Δ effects. Sixteen rats were trained in a Pavlovian and operant bidirectional contingency paradigm, using nicotine (0.3 mg/kg) and non-drug (saline) states as interoceptive cues for operant discriminative stimulus conditions (S^D and S^Δ) as well as Pavlovian FP and FN for a light-CS, either leading to a shared food pellet outcome or non-outcome. Nicotine and saline sessions were intermixed. For one group of rats (n = 8), nicotine served as an S^D for lever pressing (variable interval 60 s) and simultaneously functioned as an FN for CS-light→noUS relation on the same sessions. On intermixed sessions, saline served as the S^Δ for lever pressing (non-reinforced) and FP, during which the 8-sec light preceded delivery of the food pellet (variable time ITI = 60 s). For the other group (n = 8) nicotine served as the S^Δ (lever pressing non-reinforced) and FP for the CS, with saline serving in the reverse roles. Consecutive brief non-reinforcement tests revealed that: A) rates of lever pressing were significantly greater in S^D than S^Δ with nicotine and saline suggesting strong operant discriminative stimulus control; B) FP responding to the light CS with nicotine and saline was evident; and C) FN suppression of the CR with nicotine was not evident but weak under saline. These data suggest that nicotine can function as an interoceptive context that hierarchically can enter into concurrently opposing modulatory relations in Pavlovian and operant drug discrimination procedures.

Stimulus control of actions and habits: A role for reinforcer predictability and attention in the development of habitual behavior

Goal-directed actions are instrumental behaviors whose performance depends on the organism's knowledge of the reinforcing outcome's value. In contrast, habits are instrumental behaviors that are insensitive to the outcome's current value. Although habits in everyday life are typically controlled by stimuli that occasion them, most research has studied habits using free-operant procedures in which no discrete stimuli are present to occasion the response. We therefore studied habit learning when rats were reinforced for lever pressing on a random-interval 30-s schedule in the presence of a discriminative stimulus (S) but not in its absence. In Experiment 1, devaluing the reinforcer with taste aversion conditioning weakened instrumental responding in a 30-s S after 4, 22, and 66 sessions of instrumental training. Even extensive practice thus produced goal-directed action, not habit. Experiments 2 and 3 contrastingly found habit when the duration of S was increased from 30 s to 8 min. Experiment 4 then found habit with the 30-s S when it always contained a reinforcer; goal-directed action was maintained when reinforcers were earned at the same rate but occurred in only 50% of Ss (as in the previous experiments). The results challenge the view that habits are an inevitable consequence of repeated reinforcement (as in the law of effect) and instead suggest that discriminated habits develop when the reinforcer becomes predictable. Under those conditions, organisms may pay less attention to their behavior, much as they pay less attention to signals associated with predicted reinforcers in Pavlovian conditioning. (PsycINFO Database Record (c) 2018 APA, all rights reserved).

Inactivation of the Prelimbic Cortex Impairs the Context-Induced Reinstatement of Ethanol Seeking

Evidence indicates that drug relapse in humans is often provoked by exposure to the self-administered drug-associated context. An animal model called "ABA renewal procedure" has been used to study the context-induced relapse to drug seeking. Here, we reported a new and feasible training procedure for the ABA renewal method to explore the role of the prelimbic cortex in context-induced relapse to ethanol seeking. By using a saccharin fading technique, we trained rats to self-administer ethanol (10%). The drug delivery was paired with a discrete tone-light cue. Lever pressing was subsequently extinguished in a non-drug-associated context in the presence of the discrete cue. Rats were subsequently tested for reinstatement in contexts A or B, under extinction conditions. Ethanol-associated context induced the reinstatement of ethanol seeking and increased the expression of Fos in the prelimbic cortex. The rate of neural activation in the prelimbic cortex was 3.4% in the extinction context B and 7.7% in the drug-associated context A, as evidenced by double-labeling of Fos and the neuron-specific protein NeuN. The reversible inactivation of the neural activity in the prelimbic cortex with gamma-Aminobutyric acid (GABA) receptor agonists (muscimol + baclofen) attenuated the context-induced reinstatement of ethanol self-administration. These results demonstrated that the neuronal activation of the prelimbic cortex is involved in the context-induced reinstatement of ethanol seeking.

Factors that influence the persistence and relapse of discriminated behavior chains

Behavior chains are composed of sequences of behaviors that minimally include procurement and then consumption. This review surveys recent research from this laboratory that has examined the properties of discriminated heterogeneous behavior chains. In contrast to another review (Thrailkill and Bouton, 2016a), it discusses work examining what makes chained behavior persistent, and what makes it relapse. Results suggest that responses in a discriminated heterogeneous behavior chain may become associated, so that extinction of either one reduces the strength of the other. Evidence also suggests that the goal of the first (procurement) response may be the next (consumption) response (rather than the upcoming discriminative stimulus, a putative conditioned reinforcer, or the primary reinforcer at the end of the chain). Further studies suggest that methods that promote generalization across acquisition and extinction (partial reinforcement and delivery of noncontingent reinforcers during extinction) lead to greater persistence of the procurement response. A third set of studies analyzed the contextual control and relapse of chained behaviors. The context controls both the acquisition and extinction of chained behaviors. In addition, a separately-extinguished consumption response is renewed when returned to the context of the chain. The research expands our general understanding of the learning processes that govern instrumental behavior as well as our understanding of chains.

Renewed behavior produced by context change and its implications for treatment maintenance: A review

Behavioral treatment gains established in one setting do not always maintain in other settings. The present review examines the relevance of basic and translational research to understanding failures to maintain treatment gains across settings. Specifically, studies of the renewal effect examine how transitioning away from a treatment setting could evoke a return of undesirable behavior, rather than newly trained appropriate behavior. Studies of renewal typically arrange three phases, with a response trained and reinforced under a particular set of contextual stimuli in the first phase. Next, that response is extinguished, often under a different set of contextual stimuli. Finally, that response returns despite extinction remaining in effect upon returning to the original training context or transitioning to a novel context. Thus, removing the extinction context is sufficient to produce a recurrence of the response. The findings suggest treatment effects can become specific to the context in which the treatment was delivered. This literature offers promising methods for systematically assessing the factors contributing to treatment maintenance and improving generalization of treatment gains across contexts. Therefore, the present review suggests basic and translational research on renewal provides an empirical literature to bring greater conceptual systematization to understanding generalization and maintenance of behavioral treatment.

Inactivation of the Prelimbic Cortex Attenuates Context-Dependent Operant Responding

Operant responding in rats provides an analog to voluntary behavior in humans and is used to study maladaptive behaviors, such as overeating, drug taking, or relapse. In renewal paradigms, extinguished behavior recovers when tested outside the context where extinction was learned. Inactivation of the prelimbic (PL) region of the medial prefrontal cortex by baclofen/muscimol (B/M) during testing attenuates renewal when tested in the original acquisition context after extinction in another context (ABA renewal). Two experiments tested the hypothesis that the PL is important in context-dependent responding learned during conditioning. In the first, rats learned to lever-press for a sucrose-pellet reward. Following acquisition, animals were infused with either B/M or vehicle in the PL and tested in the acquisition context (A) and in a different context (B). All rats showed a decrement in responding when switched from Context A to Context B, but PL inactivation decreased responding only in Context A. Experiment 2a examined the effects of PL inactivation on ABC renewal in the same rats. Here, following reacquisition of the response, responding was extinguished in a new context (C). Following infusions of B/M or vehicle in the PL, responding was tested in Context C and another new context (D). The rats exhibited ACD renewal regardless of PL inactivation. Experiment 2b demonstrated that PL inactivation attenuated the ABA renewal effect in the same animals, replicating earlier results and demonstrating that cannulae were still functional. The results suggest that, rather than attenuating renewal generally, PL inactivation specifically affects ABA renewal by reducing responding in the conditioning context.SIGNIFICANCE STATEMENT Extinguished operant behavior can recover ("renew") when tested outside the extinction context. This suggests that behaviors, such as overeating or drug taking, might be especially prone to relapse following treatment. In rats, inactivation of the prelimbic cortex (PL) attenuates renewal. However, we report that PL inactivation after training attenuates responding in the context in which responding was acquired, but not in another one. A similar inactivation has no impact on renewal when testing occurs in a new, rather than the original, context following extinction. The PL thus has a more specific role in controlling contextually dependent operant behavior than has been previously reported.

The ability for cocaine and cocaine-associated cues to compete for attention

In humans, reward cues, including drug cues in individuals experiencing addiction, are especially effective in biasing attention towards them, so much so they can disrupt ongoing task performance. It is not known, however, whether this happens in rats. To address this question, we developed a behavioral paradigm to assess the capacity of an auditory drug (cocaine) cue to evoke cocaine-seeking behavior, thus distracting thirsty rats from performing a well-learned sustained attention task (SAT) to obtain a water reward. First, it was determined that an auditory cocaine cue (tone-CS) reinstated drug-seeking equally in sign-trackers (STs) and goal-trackers (GTs), which otherwise vary in the propensity to attribute incentive salience to a localizable drug cue. Next, we tested the ability of an auditory cocaine cue to disrupt performance on the SAT in STs and GTs. Rats were trained to self-administer cocaine intravenously using an Intermittent Access self-administration procedure known to produce a progressive increase in motivation for cocaine, escalation of intake, and strong discriminative stimulus control over drug-seeking behavior. When presented alone, the auditory discriminative stimulus elicited cocaine-seeking behavior while rats were performing the SAT, but it was not sufficiently disruptive to impair SAT performance. In contrast, if cocaine was available in the presence of the cue, or when administered non-contingently, SAT performance was severely disrupted. We suggest that performance on a relatively automatic, stimulus-driven task, such as the basic version of the SAT used here, may be difficult to disrupt with a drug cue alone. A task that requires more top-down cognitive control may be needed.

Context specificity of both acquisition and extinction of a Pavlovian conditioned response

It is widely held that the extinction of a conditioned response is more context specific than its initial acquisition. One proposed explanation is that context serves to disambiguate the meaning of a stimulus. Using a procedure that equated the learning histories of the contexts, we show that the memory of an appetitive Pavlovian association can be highly context specific despite being unambiguous. This result is inconsistent with predictions of the Rescorla–Wagner model of learning but in line with configural accounts of contextual control of behavior. We propose an explanatory model in which context serves to modulate the gain of associative strength and which expands upon the configural idea of unitary representations of context and conditioned stimuli.

Contextual control of operant behavior: evidence for hierarchical associations in instrumental learning

Recent research has suggested that operant responses can be weakened when they are tested in new contexts. The present experiment was therefore designed to test whether animals can learn a context-(R-O) relation. Rats were given training sessions in context A, in which one response (R1; lever pressing or chain pulling) produced one outcome (O1) and another response (R2; chain pulling or lever pressing) produced another outcome (O2) on variable interval reinforcement schedules. These sessions were intermixed with training in context B, where R1 now produced O2 and R2 produced O1. Given the arrangement, it was possible for the animal to learn two distinct R-O associations in each specific context. To test for them, rats were then given aversion conditioning with O2 by pairing its presentation with lithium-chloride-induced illness. Following the aversion conditioning, the rats were given an extinction test with both R1 and R2 available in each context. During testing, rats showed a selective suppression in each context of the response that had been paired with the reinforcer subsequently associated with illness. Rats could not have performed this way without knowledge of the R-O associations in effect in each specific context, lending support to the hypothesis that rats learn context-(R-O) associations. However, despite a complete aversion to O2, responding was not completely suppressed, leaving the possibility open that rats form context-R associations in addition to context-(R-O) associations.

Occasion setting, inhibition, and the contextual control of extinction in Pavlovian and instrumental (operant) learning

An occasion setter is a stimulus that modulates the ability of another stimulus to control behavior. A rich history of experimental investigation has identified several important properties that define occasion setters and the conditions that give rise to occasion setting. In this paper, we first consider the basic hallmarks of occasion setting in Pavlovian conditioning. We then review research that has examined the mechanisms underlying the crucial role of context in Pavlovian and instrumental extinction. In Pavlovian extinction, evidence suggests that the extinction context can function as a negative occasion setter whose role is to disambiguate the current meaning of the conditioned stimulus; the conditioning context can also function as a positive occasion setter. In operant extinction, in contrast, the extinction context may directly inhibit the response, and the conditioning context can directly excite it. We outline and discuss the key results supporting these distinctions.

Contextual control of chained instrumental behaviors

Recent studies suggest a significant role for context in controlling the acquisition and extinction of simple operant responding. The present experiments examined the contextual control of a heterogeneous behavior chain. Rats first learned a chain in which a discriminative stimulus set the occasion for a procurement response (e.g., pulling a chain), which led to a second discriminative stimulus that occasion-set a consumption response (e.g., pressing a lever) that produced a food-pellet reinforcer. Experiment 1 showed that, after separate extinction of procurement and consumption, each response increased when it was returned to the acquisition context (ABA renewal) or was tested in a new context (AAB renewal). In addition, procurement responding, but not consumption responding, was decreased by changing the context after acquisition. Experiment 2 demonstrated ABA and AAB renewal of procurement and consumption following extinction of the whole chain. This time, the context-switch after acquisition weakened both procurement and consumption. Experiment 3 found that return to the context of the behavior chain renewed a consumption response that had been extinguished separately. Finally, in Experiment 4, rats learned 2 different discriminated heterogeneous chains; consumption extinguished outside its chain was only renewed on return to a chain when it was preceded by its associated procurement response. The results suggest a role for context in the extinction of chained behavior. They also support the view that procurement is influenced by the physical context and that consumption is controlled primarily by the response that precedes it in the chain. (PsycINFO Database Record

Extinction and the associative structure of heterogeneous instrumental chains

Drug abuse, overeating, and smoking are all examples of instrumental behaviors that often involve chains or sequences of behavior. A behavior chain is minimally composed of a procurement response that is required in order for a subsequent consumption response to be reinforced. Despite the translational importance of behavior chains, few studies have attempted to understand what binds them together and takes them apart. This article surveys the development of the heterogeneous instrumental chain method and introduces recent findings that have used extinction to analyze the associative content of (what is learned in) the chain. Chained responses that are occasion-set by their own discriminative stimuli may be directly associated; extinction of the procurement response weakens its associated consumption response, and extinction of the consumption response weakens its associated procurement response. Extinction itself involves learning to inhibit the response. Extinguished chained responses are subject to renewal when they are tested either back in the acquisition context or in a new context. In addition, a consumption response that is extinguished outside its chain is renewed when returned to the context of the preceding response in the chain. Research on heterogeneous behavior chains can provide important insights into an important but often overlooked aspect of instrumental learning.

Medial prefrontal cortex involvement in the expression of extinction and ABA renewal of instrumental behavior for a food reinforcer

Instrumental renewal, the return of extinguished instrumental responding after removal from the extinction context, is an important model of behavioral relapse that is poorly understood at the neural level. In two experiments, we examined the role of the dorsomedial prefrontal cortex (dmPFC) and the ventromedial prefrontal cortex (vmPFC) in extinction and ABA renewal of instrumental responding for a sucrose reinforcer. Previous work, exclusively using drug reinforcers, has suggested that the roles of the dmPFC and vmPFC in expression of extinction and ABA renewal may depend at least in part on the type of drug reinforcer used. The current experiments used a food reinforcer because the behavioral mechanisms underlying the extinction and renewal of instrumental responding are especially well worked out in this paradigm. After instrumental conditioning in context A and extinction in context B, we inactivated dmPFC, vmPFC, or a more ventral medial prefrontal cortex region by infusing baclofen/muscimol (B/M) just prior to testing in both contexts. In rats with inactivated dmPFC, ABA renewal was still present (i.e., responding increased when returned to context A); however responding was lower (less renewal) than controls. Inactivation of vmPFC increased responding in context B (the extinction context) and decreased responding in context A, indicating no renewal in these animals. There was no effect of B/M infusion on rats with cannula placements ventral to the vmPFC. Fluorophore-conjugated muscimol was infused in a subset of rats following test to visualize infusion spread. Imaging suggested that the infusion spread was minimal and mainly constrained to the targeted area. Together, these experiments suggest that there is a region of medial prefrontal cortex encompassing both dmPFC and vmPFC that is important for ABA renewal of extinguished instrumental responding for a food reinforcer. In addition, vmPFC, but not dmPFC, is important for expression of extinction of responding for a food reinforcer. The role of the medial prefrontal cortex in renewal in the original conditioning context may depend in part on control over excitatory context-response or context-(response-outcome) relations that might be learned in acquisition. The role of the vmPFC in expression of extinction may depend on its control over inhibitory context-response or context-(response-outcome) relations that are learned in extinction.

Implications of learning theory for developing programs to decrease overeating

Childhood obesity is associated with medical and psychological comorbidities, and interventions targeting overeating could be pragmatic and have a significant impact on weight. Calorically dense foods are easily available, variable, and tasty which allows for effective opportunities to learn to associate behaviors and cues in the environment with food through fundamental conditioning processes, resulting in measurable psychological and physiological food cue reactivity in vulnerable children. Basic research suggests that initial learning is difficult to erase, and that it is vulnerable to a number of phenomena that will allow the original learning to re-emerge after it is suppressed or replaced. These processes may help explain why it may be difficult to change food cue reactivity and overeating over the long term. Extinction theory may be used to develop effective cue-exposure treatments to decrease food cue reactivity through inhibitory learning, although these processes are complex and require an integral understanding of the theory and individual differences. Additionally, learning theory can be used to develop other interventions that may prove to be useful. Through an integration of learning theory, basic and translational research, it may be possible to develop interventions that can decrease the urges to overeat, and improve the weight status of children.

Rats that sign-track are resistant to Pavlovian but not instrumental extinction

Individuals vary in the extent to which they attribute incentive salience to a discrete cue (conditioned stimulus; CS) that predicts reward delivery (unconditioned stimulus; US), which results in some individuals approaching and interacting with the CS (sign-trackers; STs) more than others (goal-trackers; GTs). Here we asked how periods of non-reinforcement influence conditioned responding in STs vs. GTs, in both Pavlovian and instrumental tasks. After classifying rats as STs or GTs by pairing a retractable lever (the CS) with the delivery of a food pellet (US), we introduced periods of non-reinforcement, first by simply withholding the US (i.e., extinction training; experiment 1), then by signaling alternating periods of reward (R) and non-reward (NR) within the same session (experiments 2 and 3). We also examined how alternating R and NR periods influenced instrumental responding for food (experiment 4). STs and GTs did not differ in their ability to discriminate between R and NR periods in the instrumental task. However, in Pavlovian settings STs and GTs responded to periods of non-reward very differently. Relative to STs, GTs very rapidly modified their behavior in response to periods of non-reward, showing much faster extinction and better and faster discrimination between R and NR conditions. These results highlight differences between Pavlovian and instrumental extinction learning, and suggest that if a Pavlovian CS is strongly attributed with incentive salience, as in STs, it may continue to bias attention toward it, and to facilitate persistent and relatively inflexible responding, even when it is no longer followed by reward.

Renewal, resurgence, and alternative reinforcement context

Resurgence, relapse induced by the removal of alternative reinforcement, and renewal, relapse induced by a change in contextual stimuli, are typically studied separately in operant conditioning paradigms. In analogous treatments of operant problem behavior, aspects of both relapse phenomena can operate simultaneously. Therefore, the purpose of this study was to examine a novel method for studying resurgence and renewal in the same experimental preparation. An alternative source of reinforcement was available during extinction for one group of rats (a typical resurgence preparation). Another group experienced an operant renewal preparation in which the extinction context was distinguished via olfactory and visual stimuli. A third group experienced alternative reinforcement delivery in the new context, a novel combination of typical resurgence and renewal preparations. Removal of alternative reinforcement and/or a change in context induced relapse relative to an extinction-only control group. When alternative reinforcement was delivered in a novel context, the alternative response was less persistent relative to when extinction of the alternative response took place in the context in which it was trained. This methodology might be used to illustrate shared (or distinct) mechanisms of resurgence and renewal, and to determine how delivering alternative reinforcement in another context may affect persistence and relapse.

Renewal after the punishment of free operant behavior

Three experiments examined the role of context in punishment learning. In Experiment 1, rats were trained to lever press for food in Context A and then punished for responding in Context B (by presenting response-contingent footshock). Punishment led to complete suppression of the response. However, when responding was tested (in extinction) in Contexts A and B, a strong renewal of responding occurred in Context A. In Experiment 2, renewal also occurred when initial reinforcement occurred in Context A, punishment occurred in Context B, and testing occurred in a new context (Context C). In both experiments, behavioral suppression and renewal were not observed in groups that received noncontingent (yoked) footshocks in Context B. In Experiment 3, 2 responses (lever press and chain pull) were separately reinforced in Contexts A and B and then punished in the opposite context. Although the procedure equated the contexts on their association with reinforcement and punishment, renewal of each response was observed when it was tested in its nonpunished context. The contexts also influenced response choice. Overall, the results suggest that punishment is specific to the context in which it is learned, and establish that its context-specificity does not depend on a simple association between the context and shock. Like extinction, punishment may involve learning to inhibit a specific response in a specific context. Implications for theories of punishment and for understanding the cessation of problematic operant behavior (e.g., drug abuse) are discussed.

Contextual control of instrumental actions and habits

After a relatively small amount of training, instrumental behavior is thought to be an action under the control of the motivational status of its goal or reinforcer. After more extended training, behavior can become habitual and insensitive to changes in reinforcer value. Recently, instrumental responding has been shown to weaken when tested outside of the training context. The present experiments compared the sensitivity of instrumental responding in rats with a context switch after training procedures that might differentially generate actions or habits. In Experiment 1, lever pressing was decremented in a new context after either short, medium, or long periods of training on either random-ratio or yoked random-interval reinforcement schedules. Experiment 2 found that more minimally trained responding was also sensitive to a context switch. Moreover, Experiment 3 showed that when the goal-directed component of responding was removed by devaluing the reinforcer, the residual responding that remained was still sensitive to the change of context. Goal-directed responding, in contrast, transferred across contexts. Experiment 4 then found that after extensive training, a habit that was insensitive to reinforcer devaluation was still decremented by a context switch. Overall, the results suggest that a context switch primarily influences instrumental habit rather than action. In addition, even a response that has received relatively minimal training may have a habit component that is insensitive to reinforcer devaluation but sensitive to the effects of a context switch.

Why behavior change is difficult to sustain

Unhealthy behavior is responsible for much human disease, and a common goal of contemporary preventive medicine is therefore to encourage behavior change. However, while behavior change often seems easy in the short run, it can be difficult to sustain. This article provides a selective review of research from the basic learning and behavior laboratory that provides some insight into why. The research suggests that methods used to create behavior change (including extinction, counterconditioning, punishment, reinforcement of alternative behavior, and abstinence reinforcement) tend to inhibit, rather than erase, the original behavior. Importantly, the inhibition, and thus behavior change more generally, is often specific to the "context" in which it is learned. In support of this view, the article discusses a number of lapse and relapse phenomena that occur after behavior has been changed (renewal, spontaneous recovery, reinstatement, rapid reacquisition, and resurgence). The findings suggest that changing a behavior can be an inherently unstable and unsteady process; frequent lapses should be expected. In the long run, behavior-change therapies might benefit from paying attention to the context in which behavior change occurs.

Mechanisms of renewal after the extinction of discriminated operant behavior

Three experiments demonstrated, and examined the mechanisms that underlie, the renewal of extinguished discriminated operant behavior. In Experiment 1, rats were trained to perform 1 response (lever press or chain pull) in the presence of one discriminative stimulus (S; light or tone) in Context A, and to perform the other response in the presence of the other S in Context B. Next, each of the original S/response combinations was extinguished in the alternate context. When the S/response combinations were tested back in the context in which they had been trained, responding in the presence of S returned (an ABA renewal effect was observed). This renewal could not be due to differential context-reinforcer associations, suggesting instead that the extinction context inhibits either the response and/or the effectiveness of the S. Consistent with the latter mechanism, in Experiment 2, ABA renewal was still observed when both the extinction and renewal contexts inhibited the same response. However, in Experiment 3, previous extinction of the response in the renewing context (occasioned by a different S) reduced AAB renewal more than did extinction of the different response. Taken together, the results suggest at least 2 mechanisms of renewal after instrumental extinction. First, extinction performance is at least partly controlled by a direct inhibitory association that is formed between the context and the response. Second, in the discriminated operant procedure, extinction performance can sometimes be partly controlled by a reduction in the effectiveness of the S in the extinction context. Renewal of discriminated operant behavior can be produced by a release from either of these forms of inhibition.

A fundamental role for context in instrumental learning and extinction

The purpose of this article is to review recent research that has investigated the effects of context change on instrumental (operant) learning. The first part of the article discusses instrumental extinction, in which the strength of a reinforced instrumental behavior declines when reinforcers are withdrawn. The results suggest that extinction of either simple or discriminated operant behavior is relatively specific to the context in which it is learned: As in prior studies of Pavlovian extinction, ABA, ABC, and AAB renewal effects can all be observed. Further analysis supports the idea that the organism learns to refrain from making a specific response in a specific context, or in more formal terms, an inhibitory context-response association. The second part of the article then discusses research suggesting that the context also controls instrumental behavior before it is extinguished. Several experiments demonstrate that a context switch after either simple or discriminated operant training causes a decrement in the strength of the response. Over a range of conditions, the animal appears to learn a direct association between the context and the response. Under some conditions, it can also learn a hierarchical representation of context and the response-reinforcer relation. Extinction is still more context-specific than conditioning, as indicated by ABC and AAB renewal. Overall, the results establish that the context can play a significant role in both the acquisition and extinction of operant behavior.

Behavioral and neurobiological mechanisms of extinction in Pavlovian and instrumental learning

This article reviews research on the behavioral and neural mechanisms of extinction as it is represented in both Pavlovian and instrumental learning. In Pavlovian extinction, repeated presentation of a signal without its reinforcer weakens behavior evoked by the signal; in instrumental extinction, repeated occurrence of a voluntary action without its reinforcer weakens the strength of the action. In either case, contemporary research at both the behavioral and neural levels of analysis has been guided by a set of extinction principles that were first generated by research conducted at the behavioral level. The review discusses these principles and illustrates how they have informed the study of both Pavlovian and instrumental extinction. It shows that behavioral and neurobiological research efforts have been tightly linked and that their results are readily integrated. Pavlovian and instrumental extinction are also controlled by compatible behavioral and neural processes. Since many behavioral effects observed in extinction can be multiply determined, we suggest that the current close connection between behavioral-level and neural-level analyses will need to continue.

Context change and associative learning

This article reviews the effects of changing the background context on performance in associative learning tasks in humans and animals. The findings are complementary and consistent over animal conditioning (Pavlovian and instrumental learning) and human predictive learning and memory paradigms. In many cases, a context change after learning can have surprisingly little disruptive influence on performance. Extinction, or retroactive interference treatments more generally, is more context-specific than the initial learning. Contexts become important if the participant is exposed to any of several treatments that involve prediction error, which may serve to increase attention to the context. Contexts also become important if they are given predictive or informational value. Studies of instrumental (operant) learning are further consistent with the idea that the context might also influence affordances that support voluntary actions. Context switch effects are not universal, but mainly occur when certain attention and perception processes can come into play. WIREs Cogn Sci 2013, 4:237-244. doi: 10.1002/wcs.1225 This article is categorized under: Psychology > Learning.