Discriminated response and incentive processes in operant conditioning: a two-factor model of stimulus control

Lateral orbitofrontal cortex integrates predictive information across multiple cues to guide behavior

Individuals are often faced with multiple cues that concurrently predict the same outcome, and combining these predictions may benefit behavior. Previous work has studied the neural basis of decision-making, predominantly using isolated sensory stimuli, and so the mechanisms that allow us to leverage multiple cues remain unclear. In two experiments, we used neuroimaging and network-targeted brain stimulation to probe how the brain integrates outcome predictions to guide adaptive behavior. We identified neural signatures of outcome integration in the lateral orbitofrontal cortex (OFC), where concurrently presented cues evoke stronger pattern-based representations of expected outcomes. Moreover, perturbing lateral OFC network activity impairs subjects' ability to leverage predictions from multiple cues to facilitate responding. Intriguingly, we found similar behavioral and brain mechanisms for reward-predicting cues and for cues predicting the absence of reward. These findings highlight a causal role for the lateral OFC in utilizing outcome predictions from multiple cues to guide behavior.

Discriminative stimuli are sufficient for incubation of cocaine craving

In abstinent drug addicts, cues formerly associated with drug-taking experiences gain relapse-inducing potency ('incubate') over time. Animal models of incubation may help develop treatments to prevent relapse, but these models have ubiquitously focused on the role of conditioned stimuli (CSs) signaling drug delivery. Discriminative stimuli (DSs) are unique in that they exert stimulus-control over both drug taking and drug seeking behavior and are difficult to extinguish. For this reason, incubation of the excitatory effects of DSs that signal drug availability, not yet examined in preclinical studies, could be relevant to relapse prevention. We trained rats to self-administer cocaine (or palatable food) under DS control, then investigated DS-controlled incubation of craving, in the absence of drug-paired CSs. DS-controlled cocaine (but not palatable food) seeking incubated over 60 days of abstinence and persisted up to 300 days. Understanding the neural mechanisms of this DS-controlled incubation holds promise for drug relapse treatments.

Moving Beyond Reinforcement and Response Strength

Behavior analysis has often simultaneously depended upon and denied an implicit, hypothetical process of reinforcement as response strengthening. I discuss what I see as problematic about the use of such an implicit, possibly inaccurate, and likely unfalsifiable theory and describe issues to consider with respect to an alternative view without response strengthening. In my take on such an approach, important events (i.e., "reinforcers") provide a means to measure learning about predictive relations in the environment by modulating (i.e., inducing) performance dependent upon what is predicted and the relevant motivational mode or behavioral system active at that time (i.e., organismic state). Important events might be phylogenetically important, or they might acquire importance by being useful as signals for guiding an organism to where, when, or how currently relevant events might be obtained (or avoided). Given the role of learning predictive relations in such an approach, it is suggested that a potentially useful first step is to work toward formal descriptions of the structure of the predictive relations embodied in common facets of operant behavior (e.g., response-reinforcer contingencies, conditioned reinforcement, and stimulus control). Ultimately, the success of such an approach will depend upon how well it integrates formal characterizations of predictive relations (and how they are learned without response strengthening) and the relevant concomitant changes in organismic state across time. I also consider how thinking about the relevant processes in such a way might improve both our basic science and our technology of behavior.

A comparison of therapies for the treatment of drug cues: counterconditioning vs. extinction in male rats

Although extinction has been used as a treatment to reduce the power of drug cues, a better method is needed. Research with traditional reinforcers has shown that counterconditioning--pairing an appetitive cue with an aversive stimulus--can suppress cue-controlled behavior. The present experiment compared the counterconditioning and extinction of cocaine cues. Male rats were first trained to self-administer cocaine during a light cue. In the second phase, the light was paired with footshock in the counterconditioning group. The extinction group was treated similarly, except light presentations did not end in footshock. Counterconditioning suppressed cocaine seeking to a greater extent than extinction while the counterconditioning treatment was actively administered. On a subsequent stimulus compounding test, where footshock was discontinued and the light was presented simultaneously with an untreated cocaine cue (a tone), suppressive effects of counterconditioning were evident during the early portion of the test but not during later trials. Overall, results of the present experiment suggest that counterconditioning produces only temporarily suppressive effects on cue-controlled cocaine seeking. Methods for directly weakening the cue-drug association (e.g., "deepened extinction") may prove to be more useful drug cue treatments.

Extinguished cocaine cues increase drug seeking when presented simultaneously with a non-extinguished cocaine cue

BACKGROUND

Previous research with non-drug reinforcers has shown that simultaneously presenting (compounding) an extinguished cue with another cue formerly associated with the same reinforcer can increase rates of cue-controlled behavior. The present study investigated whether an extinguished cocaine cue would energize cocaine seeking when presented simultaneously with another cocaine cue. This study also investigated whether extinction could be enhanced by subjecting an extinguished cocaine cue to further extinction after administration of reinstating injections of cocaine.

METHODS

Rats were first trained to self-administer cocaine in the presence of three different cues. Then, one of the cues was subjected to the standard extinction treatment. Another cue was subjected to a modified extinction treatment where additional extinction sessions were preceded by non-contingent cocaine injections. The third cue was not extinguished.

RESULTS

The cue subjected to standard extinction ceased to control cocaine seeking when presented alone, but significantly increased cocaine seeking when compounded with the non-extinguished cocaine cue. The cocaine cue subjected to the modified extinction treatment also significantly increased cocaine seeking occasioned by the non-extinguished cocaine cue.

CONCLUSIONS

Extending results of previous studies involving non-drug stimuli, the present study showed that extinguished cocaine cues can enhance cocaine seeking when compounded with other cocaine cues. These results illustrate the persistence of drug cues in controlling behavior despite extinction and highlight the need for developing treatments that eliminate this residual energizing capacity that survives extinction.

Generalization peak shift in rats under conditions of positive reinforcement and avoidance

Rats were trained to discriminate between two click frequencies. One frequency was associated with either variable-interval food reinforcement (Experiment 1) or free-operant avoidance (Experiment 2). The other frequency was associated with the absence of food in Experiment 1 and the absence of shock in Experiment 2. On a click frequency generalization test, the rats in both experiments showed positive peak shift with the shape of the relative gradients being very similar. This is the first reported instance of peak shift in rats when responding was maintained by an avoidance contingency. Nondifferentially trained controls showed that this shift was due exclusively to associative processes, with nonassociative stimulus factors in themselves apparently making no contribution to increased rates at particular stimulus values. These results show the comparability of appetitive and aversive control and support the position that gradient differences do not result from approach versus avoidance per se.

Within-subject reversibility of discriminative function in the composite-stimulus control of behavior

According to the composite-stimulus control model (Weiss, 1969, 1972b), an individual discriminative stimulus (S(D)) is composed of that S(D)'s on-state plus the off-states of all other relevant S(D)s. The present experiment investigated the reversibility of composite-stimulus control. Separate groups of rats were trained to lever-press for food whenever a tone or a light S(D) was present. For one group, the nonreinforced S(Delta) condition was tone-and-light absence (T+L). Tone-plus-light (T+L) was S(Delta) in the other group. On a "stimulus compounding" test that recombined composite elements, maximum responding occurred to that composite consisting only of elements occasioning response increase. That was T+L for the group trained with T+L as S(Delta) and T+L for the group trained with T+L as S(Delta). The S(Delta) composite was next reversed over groups in Phase 2. In Phase 2 tests, maximum responding that was comparable in magnitude to that of Phase 1 was again controlled by the composite consisting only of elements most recently occasioning response increase-whether T+L or T+L. The inhibitory conditioning history of both composite-elements currently occasioning responding did not weaken the summative effect. These results confirm and extend Weiss's composite-stimulus control model, and demonstrate that such control is fully reversible. We discuss how translating conditions of the stimulus-compounding paradigm to a composite continuum creates a functional and logical connection to intradimensional control measured through stimulus generalization, reducing the number of different behavioral phenomena requiring unique explanations.

A comparison of the effects of discriminative and Pavlovian inhibitors and excitors on instrumental responding

In two experiments, the effects of Pavlovian or discriminative conditioned inhibitors on operant responding were investigated in rats. Experiment 1 found that a Pavlovian conditioned inhibitor for food suppressed food-reinforced lever pressing more than a non-differentially trained control stimulus did. Experiment 2 demonstrated that an operant discriminative inhibitor produced greater suppression of lever pressing than a Pavlovian conditioned inhibitor. Experiment 2 also found that compounding an operant discriminative stimulus (SD) for food-reinforced responding with another SD for food-reinforced responding resulted in more additive summation than when an SD was compounded with a Pavlovian conditioned excitor for food. The results of these experiments support two-factor theories that postulate that incentive and response discriminative processes summate algebraically when the processes are inhibitory or excitatory.

Stimulus control and compounding with ambient odor as a discriminative stimulus on a free-operant baseline

Previous experiments have demonstrated that the simultaneous presentation of independently established discriminative stimuli can control rates of operant responding substantially higher than the rates occasioned by the individual stimuli. This "additive summation" phenomenon has been shown with a variety of different reinforcers (e.g., food, water, shock avoidance, cocaine, and heroin). Discriminative stimuli previously used in such studies have been limited to the visual and auditory sensory modalities. The present experiment sought to (1) establish stimulus control on a free-operant baseline with an ambient olfactory discriminative stimulus, (2) compare olfactory control to that produced with an auditory discriminative stimulus, and (3) determine whether compounding independently established olfactory and auditory discriminative stimuli produces additive summation. Rats lever pressed for food on a variable-interval schedule in the presence of either a tone or an odor, with comparable control developed to each stimulus. In the absence of these stimuli responding was not reinforced. During stimulus compounding tests, the tone-plus-odor compound occasioned more than double the responses occasioned by either the tone or odor presented individually. Thus, the current study (1) established stimulus control with an ambient olfactory discriminative stimulus in a traditional free-operant setting and (2) extended the generality of stimulus-compounding effects by demonstrating additive summation when olfactory and auditory discriminative stimuli were presented simultaneously.

Ethanol-maintained responding of rats is more resistant to change in a context with added non-drug reinforcement

Alternative non-drug reinforcers reliably decrease drug-maintained responding in self-administration procedures. Studies of the resistance to change of food-maintained behavior, however, have found that responding in the presence of a stimulus associated with an alternative reinforcer is more resistant to disruption. This increase in persistence occurs despite lower response rates when the alternative reinforcer is present. The present experiment examined if, in addition to decreasing response rates, an alternative non-drug reinforcer also increases the persistence of drug-maintained responding. Rats self-administered oral ethanol in a multiple schedule of reinforcement in which responding was reinforced in two components signaled by different stimuli. In one component, response-independent food was delivered in addition to the earned ethanol. The effects of the alternative food reinforcer on response rates and resistance to extinction in the two components were examined. As in previous experiments on the resistance to change of food-maintained operant behavior, response rates were lower, but more resistant to extinction in the presence of the stimulus associated with the alternative reinforcer. These findings suggest that all the reinforcers obtained in a context in which drugs are consumed may contribute to the persistence of drug seeking in that context. This increase in persistence may occur even if the alternative reinforcers interfere with drug seeking.

Stimuli produced by observing responses make rats' ethanol self-administration more resistant to price increases

RATIONALE

Observing responses bring sensory receptors into contact with environmental stimuli. In the observing-response procedure, periods in which an operant response (e.g. pressing a lever) is reinforced by drug deliveries alternate with periods in which this response is never reinforced (i.e. extinction). These alternating periods of drug availability versus extinction are not signaled. Observing responses (i.e. presses on a second lever) produce brief stimuli signaling whether drug is available or not for responses on the first lever. Little is known about how parameters of the drug reinforcer affect drug-stimulus observing.

OBJECTIVES

The effects of changes in the unit price (responses/reinforcer magnitude) of self-administered ethanol on rats' observing were examined. Also, the effects of an observing-response-produced ethanol stimulus on ethanol consumption were examined by comparing consumption during signaled and unsignaled periods of ethanol availability.

METHODS

Rats self-administered oral ethanol in the observing-response procedure. The unit price of ethanol in the observing-response procedure was increased by increasing the response requirement for ethanol across conditions.

RESULTS

Observing and response rates on the ethanol lever increased and then decreased with increases in the unit price of ethanol. However, ethanol-lever responding and ethanol consumption during periods when ethanol was available were less sensitive to increases in price when the observing-response-produced ethanol stimulus was present.

CONCLUSIONS

Observing varies as an orderly function of unit price of a drug reinforcer, and drug stimuli produced by observing responses can make drug consumption less sensitive to increases in price. This procedure may provide an animal model of both attending to drug stimuli and the resultant effects of these stimuli on drug taking.

Stimulus control of cocaine self-administration

Environmental stimuli that set the occasion wherein drugs are acquired can "trigger" drug-related behavior. Investigating the stimulus control of drug self-administration in laboratory animals should help us better understand this aspect of human drug abuse. Stimulus control of cocaine self-administration was generated here for the first time using multiple and chained schedules with short, frequently-alternating components--like those typically used to study food-maintained responding. The procedures and results are presented along with case histories to illustrate the strategies used to produce this stimulus control. All these multicomponent schedules contained variable-interval (VI) components as well as differential-reinforcement-of-other-behavior (DRO) or extinction components. Schedule parameters and unit dose were adjusted for each rat to produce stable, moderate rates in VI components, with minimal postreinforcement (infusion) pausing, and response cessation in extinction and DRO components. Whole-body drug levels on terminal baselines calculated retrospectively revealed that all rats maintained fairly stable drug levels (mean, 2.3 to 3.4 mg/kg) and molar rates of intake (approximately 6.0 mg/kg/hr). Within this range, no relation between local VI response rates and drug level was found. The stimulus control revealed in cumulative records was indistinguishable from that achieved with food under these schedules, suggesting that common mechanisms may underlie the control of cocaine- and food-maintained behavior.

Effects of compounding drug-related stimuli: escalation of heroin self-administration

Previous experiments have demonstrated that presenting independently established discriminative stimuli in compound can substantially increase operant responding maintained by food reinforcement or shock avoidance. Recently, this phenomenon was also shown to occur with cocaine self-administration. The present study further assessed the generality of these stimulus-compounding effects by systematically replicating them with heroin self-administration. Rats' nose-poke responses produced intravenous heroin (0.025 mg/kg per infusion) on a variable-ratio schedule when either a tone or a light was present. In the absence of these stimuli, responding was not reinforced. Once discriminative control by the tone and light had been established, the stimuli were presented in compound under extinction (with heroin discontinued) or maintenance conditions (with heroin available during test-stimulus presentations). In extinction, the tone-light compound increased responding approximately threefold compared to tone or light alone. Under maintenance conditions, compounding increased heroin intake approximately twofold. These effects closely matched those obtained earlier with cocaine. This consistency across pharmacological classes and across drug and nondrug reinforcers further confirms that (a) self-administered drugs support conditioning and learning in a manner similar to that supported by other reinforcers; and (b) multiple drug-related cues interact in lawful and predictable ways to affect drug seeking and consumption.

Cocaine self-administration increased by compounding discriminative stimuli

Presenting independently established discriminative stimuli in compound can substantially increase response rates under food and shock-avoidance schedules. To determine whether this effect extends to drug self-administration, rats were trained to press a lever to receive cocaine intravenously. A tone and a light were independently established as discriminative stimuli for cocaine self-administration, then presented in combination in a stimulus-compounding test. Compared to tone and light alone, the tone-plus-light compound stimulus increased responding approximately three-fold when cocaine was withheld during testing, and it increased drug intake approximately two-fold when cocaine was made available during testing. Compounding did not increase responding after training in a truly random control condition where tone and light were presented uncorrelated with the availability of cocaine. The results obtained with this animal model of drug abuse define conditions under which combinations of environmental stimuli might substantially increase human drug use.

Reversibility of single-incentive selective associations

Rats were trained to press a lever in the presence of a tone-light compound stimulus and not to press in its absence. In each of two experiments, schedules were designed to make the compound a conditioned punisher for one group and a conditioned reinforcer for the other. In Experiment 1, one group's responding produced food in the presence of the compound but not in its absence. The other group's responding terminated the compound stimulus, and food was presented only in its absence. When tone and light were later presented separately, light controlled more responding than did tone in the former group, but tone gained substantial control in the latter. The same effects were also observed within subjects when the training schedules were switched over groups. In Experiment 2, two groups avoided shock in the presence of the compound stimulus. In the absence of the compound, one group was not shocked, and the other received both response-independent and response-produced shock. When tone and light were presented separately, the former group's responding was mainly controlled by tone, but the latter group's responding was almost exclusively controlled by light. These effects were also observed within subjects when the training schedules were switched over groups. Thus, these single-incentive selective association effects (appetitive in Experiment 1 and aversive in Experiment 2) were completely reversible. The schedules in which the compound should have been a conditioned reinforcer consistently produced visual control, and auditory control increased when the compound should have become a conditioned punisher. Currently accepted accounts of selective associations based on affinities between shock and auditory stimuli and between food and visual stimuli (i.e., stimulus-reinforcer interactions) do not adequately address these results. The contingencies of reinforcement most recently associated with the compound and with its absence, rather than the nature of the reinforcer, determined whether auditory or visual stimulus control developed.

Selective associations produced solely with appetitive contingencies: the stimulus-reinforcer interaction revisited

In studies reporting stimulus-reinforcer interactions in traditional conditioning paradigms, when a tone-light compound was associated with food the light gained stimulus control, but when the compound was paired with shock avoidance the tone gained control. However, the physical nature of the reinforcer-related events (food vs. shock) presented in the presence of the tone-light compound was always confounded with the conditioned hedonic value of the compound's presence relative to its absence. When the compound was paired with shock, its presence was negative relative to its absence (which was shock-free). In contrast, when the compound was paired with food, its presence was positive relative to its absence (which was food-free). The present experiment dealt with this confounding effect by conditioning a tone-light compound to be positive or negative, relative to its absence, solely with food reinforcement. One group of rats received food for responding in the presence of the tone-light compound and no food in its absence. The other group also responded in the presence of the compound, but received food only in its absence. These rats were trained on a chained schedule in which responding in the presence of the tone-light compound produced a terminal link signaled by the absence of the compound; responding ceased in the terminal link because it delayed food delivery. In a test session to assess stimulus control by the elements of the compound, tone and light were presented separately under extinction conditions. Rats that had been exposed to a positive correlation between food and the compound emitted almost double the responses in the presence of the light as in the presence of the tone. In comparison, rats that had been exposed to a negative correlation emitted only two thirds as many responses in the presence of the light as in the presence of the tone. Because this selective association was produced using only food, it appears that the contingencies under which a reinforcer is presented, rather than (or as well as) its physical properties, can generate the selective associations previously attributed to "stimulus-reinforcer interactions." This could mean that regardless of the class of reinforcer that ultimately maintains responding (appetitive or aversive), the contingency-generated hedonic value of the compound stimulus may influence the dominant modality of stimulus control.

Generalization peak shift for autoshaped and operant key pecks

Pigeons acquired discriminated key pecking between 528- and 540-nm stimuli by either a response-reinforcer (operant group) or a stimulus-reinforcer (autoshaped group) contingency, with other training-schedule parameters comparable over groups. For the birds in the operant group, key pecks intermittently produced grain in the presence of one hue on the key (positive stimulus) but not in the other (negative stimulus). For the birds in the autoshaped group, pecking emerged when grain was intermittently presented independently of key pecking during one key color but was not presented during the other key color. Two independent contingency assays, peck-location comparisons and elimination of differences in reinforcement rate, confirmed the effectiveness of the two training procedures in establishing operant or respondent control of key pecking. After reaching a 10:1, or better, discrimination ratio between key pecks during the two key colors, the birds received a wavelength generalization test. Criterion baseline key-peck rates were comparable for operant and autoshaped groups prior to testing. On the generalization test, performed in extinction, all birds pecked most at a stimulus removed from the positive training stimulus in the direction away from the negative stimulus. In testing, autoshaped "peak" rates (24.5 to 64.9 pecks per minute) were from 33% to 80% higher than rates in the presence of the training stimuli. Respondent peak shift rarely has been reported heretofore, and never this consistently and robustly. These results further confirm the similarity of perceptual processing in classical and operant learning. They are discussed in terms of Spence's gradient-interaction theory and Weiss' (1978) two-process model of stimulus control.

The integration of habits maintained by food and water reinforcement through stimulus compounding

In Experiment 1, a light and a tone were correlated independently with water reinforcement of bar pressing by rats. With different naive subjects in Experiment 2, one of these stimuli was correlated with food and the other with water reinforcement (counterbalanced). In both experiments the absence of tone and light signaled extinction. Tests of stimulus-reinforcer independence in Experiment 2 indicated that tone and light controlled behavior whose rate was specifically affected by deprivation state. In the stimulus-compounding tests of both experiments, response rates were higher to tone-plus-light than to tone or light presented alone (additive summation). This is the first report of additive summation produced through compounding stimuli paired with different reinforcers. The results are discussed in the context of the effects of incentive motivation on operant performance.

Incentive processes and the peak shift

Intradimensional operant discrimination schedules were employed, which eliminated the covariation of response and reinforcement rates that are found on most operant baselines. In Phase 1, one keylight (S(1)) controlled an increase in pigeons' treadle pressing, relative to another keylight (S(2)), while being correlated with a decrease in frequency of reinforcement. In Phase 2 both treadle pressing and reinforcement increased in the presence of one keylight, relative to the second. In Phase 1 the relatively flat treadle-press generalization gradients peaked at S(1), whereas the peaks of those in Phase 2 were shifted from S(1) in a direction away from S(2). It was postulated that these positive and negative stimulus-reinforcement contingencies influence the likelihood of obtaining peak shift through the operation of a classically conditioned "central motive state." How response-reinforcement and stimulus-reinforcement contingencies might contribute to the development of inhibitory effects of S(2) is discussed. Autoshaped key pecking also was produced by these procedures. During manipulations of stimuli, the gradients obtained for autoshaped key pecking were narrow and sharply peaked at the food-correlated stimulus (S(2)) in Phase 1. This failure to obtain peak shift for an elicited response suggests a difference in discriminative processes operating in classical and instrumental learning.

An investigation of peak shift and behavioral contrast for autoshaped and operant behavior

Instrumental treadle press and nonreinforced key peck responses were monitored during discrimination training and generalization testing in pigeons on positive and negative reinforcement schedules. In Experiment 1, six pigeons pressed a treadle for food on a multiple variable-interval extinction schedule. In Experiment 2, three pigeons pressed a treadle to avoid shock on a multiple free-operant avoidance extinction schedule. Different color keylights signaled S+ and S- components. Some positive behavioral contrast occurred during discrimination training, but the effect was small. Pecking occurred to the S+ keylight in Experiment 1 but not in Experiment 2. On stimulus generalization tests, all subjects displayed a positive peak shift when pressing the treadle for food or to avoid shock. However, peak shift was not found for nonreinforced "autopecks" on the stimulus key, although an area shift was observed in Experiment 1. This is the first demonstration of peak shift for pigeons pressing treadles and the only reliable demonstration of peak shift when negative reinforcement maintained responding. These results, in combination with previous demonstrations of peak shift for rats pressing levers and pigeons pecking keys, indicate that peak shift is a general by-product of operant discrimination learning, since it occurs across a variety of the organisms, responses, and reinforcers.