Conditioned reinforcement and information theory reconsidered

Evaluating functions of praise for children diagnosed with autism spectrum disorder

We assessed whether novel praise statements could be used to (a) maintain and increase responses with existing reinforcement histories and (b) teach a previously untaught response among children diagnosed with autism spectrum disorder across two experiments. During response-stimulus pairing, two responses resulted in preferred edibles but only one also produced a praise statement. In the absence of edibles, the response continuing to produce praise tended to persist more. Next, reversing the praise contingency tended to increase the other response. However, in no case did contingent delivery of those same praise statements result in the acquisition of untaught responses. These findings suggest that conditioning praise statements could serve different functions (antecedent or consequence) depending on the reinforcement history for particular responses.

Effects of inter-trial interval on sign-tracking and conditioned reinforcer efficacy in female rats

Previous nonhuman studies have reported that sign-tracking to a conditioned stimulus (CS) is increased when the intertrial interval (ITI) duration is increased. Separate studies indicate that individual differences in sign-tracking (vs. goal-tracking) at a fixed ITI (and CS duration) is predictive of the conditioned reinforcer efficacy of the CS. The present study evaluates, for the first time, if increasing the ITI increases rats' sign-tracking and the conditioned reinforcing efficacy of the CS. Forty-five female rats were randomly assigned to one of three groups that completed appetitive Pavlovian training with ITIs of 14, 24, or 96 s. Subsequently, they completed tests of conditioned reinforcement. Replicating previous findings, longer ITIs increased sign-tracking to a lever-CS and, extending the literature, conditioned reinforcer efficacy of that CS was highest at the longest ITI used during Pavlovian training. Implications for behavioral interventions using conditioned reinforcement are discussed.

Pavlovian learning and conditioned reinforcement

Conditioned reinforcers are widely used in applied behavior analysis. Basic research evidence reveals that Pavlovian learning plays an important role in the acquisition and efficacy of new conditioned-reinforcer functions. Thus, a better understanding of Pavlovian principles holds the promise of improving the efficacy of conditioned reinforcement in applied research and practice. This paper surveys how (and if) Pavlovian principles are presented in behavior-analytic textbooks; imprecisions and knowledge gaps within contemporary Pavlovian empirical findings are highlighted. Thereafter, six practical principles of Pavlovian conditioning are presented along with empirical support and knowledge gaps that should be filled by applied and translational behavior-analytic researchers. Innovative applications of these principles are outlined for research in language acquisition, token reinforcement, and self-control.

Paradoxical choice and the reinforcing value of information

Signals that reduce uncertainty can be valuable because well-informed decision-makers can better align their preferences to opportunities. However, some birds and mammals display an appetite for informative signals that cannot be used to increase returns. We explore the role that reward-predictive stimuli have in fostering such preferences, aiming at distinguishing between two putative underlying mechanisms. The 'information hypothesis' proposes that reducing uncertainty is reinforcing per se, somewhat consistently with the concept of curiosity: a motivation to know in the absence of tractable extrinsic benefits. In contrast, the 'conditioned reinforcement hypothesis', an associative account, proposes asymmetries in secondarily acquired reinforcement: post-choice stimuli announcing forthcoming rewards (S⁺) reinforce responses more than stimuli signalling no rewards (S^-) inhibit responses. In three treatments, rats faced two equally profitable options delivering food probabilistically after a fixed delay. In the informative option (Info), food or no food was signalled immediately after choice, whereas in the non-informative option (NoInfo) outcomes were uncertain until the delay lapsed. Subjects preferred Info when (1) both outcomes were explicitly signalled by salient auditory cues, (2) only forthcoming food delivery was explicitly signalled, and (3) only the absence of forthcoming reward was explicitly signalled. Acquisition was slower in (3), when food was not explicitly signalled, showing that signals for positive outcomes have a greater influence on the development of preference than signals for negative ones. Our results are consistent with an elaborated conditioned reinforcement account, and with the conjecture that both uncertainty reduction and conditioned reinforcement jointly act to generate preference.

Motivating Operations and Negative Reinforcement

The motivating operations concept has improved the precision of our approach to analyzing behavior; it serves as a framework for classifying events that alter the reinforcing and punishing effectiveness of other events. Nevertheless, some aspects of the concept are seriously flawed, thereby limiting its utility. We contend in this article that the emphasis it places on the onset of some stimuli (putative motivating operations) making their offset a reinforcer in the absence of a learning history (i.e., in the case of unconditioned motivating operations), or because of such a history (i.e., in the case of reflexive conditioned motivating operations), is of no value in predicting or controlling behavior. It is unfortunate that this pseudo-analysis has been widely accepted, which has drawn attention away from actual motivating operations that are relevant to negative reinforcement, and led to conceptually flawed explanations of challenging human behaviors that are escape-maintained. When used appropriately, the motivating operations concept can help to clarify the conditions under which a stimulus change (in particular, stimulus termination) will function as a negative reinforcer. From both a theoretical and a practical perspective, rethinking the application of the motivating operations concept to negative reinforcement is advantageous. Herein, we explore the implications of doing so with the aim of encouraging relevant research and improving the practice of applied behavior analysis.

Reducing impulsive choice VII: effects of duration of delay-exposure training

Impulsive choice is related to substance use disorders, obesity, and other behaviors that negatively impact human health. Reducing impulsive choice may prove beneficial in ameliorating these maladaptive behaviors. Preclinical research in rats indicates that one reliable method for producing large and lasting reductions in impulsive choice is delay-exposure (DE) training. In all six of the prior DE-training experiments, rats were given extensive experience (~ 120 training sessions) with a delayed reinforcement contingency. The present experiment evaluated if similar large and lasting reductions in impulsive choice could be achieved with less training. The duration of DE training between groups of male Wistar rats was 0 sessions (training ended after a lever-pressing acquisition criterion was met), 30, 60, or 120 sessions. Comparison groups were given the same durations of training with immediate reinforcement. A post-training assessment of impulsive choice was completed using an increasing-delay procedure. For rats assigned to the 60-session condition, impulsive choice was reassessed at a 120-day follow-up. DE training reduced impulsive choice but, contrary to expectation, reductions in impulsive choice did not increase with DE-training duration (no significant training-duration by group interaction). Importantly, 60 sessions of DE training produced reductions in impulsive choice that were comparable to prior published findings and this effect remained significant at the 120-day follow-up. Procedural elements that may be responsible for the DE-training effect, and how they could be improved in future experiments, are discussed.

Ethanol-paired stimuli can increase reinforced ethanol responding

While ethanol-paired stimuli are frequently postulated to increase drinking motivation and thus increase ethanol responding and precipitate relapse, no study has demonstrated increases in ethanol-reinforced responding following presentation of an ethanol-paired stimulus that had not previously been part of a contingent relationship. Previous studies have shown that food-paired stimuli can increase food responding that is at low rates and increase food consumption in food-sated rats. In Experiment 1, we show that an ethanol-paired stimulus can increase ethanol responding that is at low levels late in the experimental session, presumably due to satiation. However, these increases may have resulted from either associative or non-associative mechanisms. In Experiment 2, we compared the effects of an ethanol-paired stimulus to those of the same stimulus in a Truly-Random-Control group. In a Truly-Random-Control, the stimulus and ethanol each are presented on independent random schedules, and thus any differences between the effects of the stimulus in the experimental and control groups is likely attributable to the association between the stimulus and ethanol. The stimulus increased ethanol-reinforced responding in both the experimental and control groups, but these increases were greater in the experimental than the control group. Thus, both stimulus-change and the pairing of the stimulus with ethanol may result in increases in ethanol-reinforced responding.

Number and time in acquisition, extinction and recovery

We measured rate of acquisition, trials to extinction, cumulative responses in extinction, and the spontaneous recovery of anticipatory hopper poking in a Pavlovian protocol with mouse subjects. We varied by factors of 4 number of sessions, trials per session, intersession interval, and span of training (number of days over which training extended). We find that different variables affect each measure: Rate of acquisition [1/(trials to acquisition)] is faster when there are fewer trials per session. Terminal rate of responding is faster when there are more total training trials. Trials to extinction and amount of responding during extinction are unaffected by these variables. The number of training trials has no effect on recovery in a 4-trial probe session 21 days after extinction. However, recovery is greater when the span of training is greater, regardless of how many sessions there are within that span. Our results and those of others suggest that the numbers and durations and spacings of longer-duration "episodes" in a conditioning protocol (sessions and the spans in days of training and extinction) are important variables and that different variables affect different aspects of subjects' behavior. We discuss the theoretical and clinical implications of these and related findings and conclusions-for theories of conditioning and for neuroscience.

Suboptimal choice and initial-link requirement

Pigeons (n = 14) were trained in a concurrent-chains suboptimal choice procedure that tested the effect of an increased ratio requirement in the initial links. Fixed-ratio 1 and 25 conditions were manipulated within subjects in a counterbalanced order. In all conditions, distinct terminal-link stimuli on a suboptimal alternative signaled either primary reinforcement (20% of the time) or extinction (80% of the time). On an optimal alternative, two distinct terminal-link stimuli each signaled a 50% chance of primary reinforcement. Preference for the suboptimal alternative was significantly attenuated, and in some birds completely reversed, by the larger response requirement irrespective of condition order. This larger response requirement also generated a notable increase in between-subject variability. A measure of cumulative choice responding is introduced to mitigate the problems associated with traditional session averages. Ordinal predictions of some current theories of suboptimal choice are also considered in light of the results.

Effects of rat strain and method of inducing ethanol drinking on Pavlovian-Instrumental-Transfer with ethanol-paired conditioned stimuli

Ethanol-paired conditioned stimuli (CSs) are widely thought to invigorate ethanol responding, and thus, precipitate relapse to drinking. However, preclinical studies investigating this issue using Pavlovian-Instrumental-Transfer (PIT) procedures have had mixed results, with some studies finding PIT while others did not. The studies failing to show PIT used Lewis rats and induced ethanol drinking using a post-prandial drinking procedure. The present experiments examined whether either of these two variables influenced the magnitude of PIT observed. In the first experiment, ethanol drinking in Lewis rats was induced using either sucrose fading or post-prandial drinking. In the second experiment, ethanol drinking was induced using post-prandial drinking in either Long-Evans Hooded or Lewis rats. In both experiments, rats were trained to respond for ethanol under a random interval schedule. Subsequently with the lever removed, 2-min light presentations were paired with ethanol deliveries. Finally, with the lever returned, the effect of light presentations on responding was tested while responding was in extinction. Light presentations similarly affected responding in Lewis rats regardless of the method of drinking induction. Likewise, light presentations similarly affected responding in both Lewis and Long-Evans Hooded rats. Neither ethanol induction method nor rat strain affected the magnitude of PIT observed, and thus, neither likely explains previous failures to observe PIT with ethanol-maintained behavior.

Effects of schedule type and terminal-link duration on choice in concurrent chains

Four pigeons responded in a three-component concurrent chains procedure in which the terminal-link schedules were either both fixed-interval (FI FI), both variable-interval (VI VI), both mixed-interval (MI MI) or variable-interval fixed-interval (VI FI). Across components within sessions, overall terminal-link duration was varied while schedule types were varied across conditions. For the conditions with homogeneous schedules, the strongest preference was obtained with FI FI, intermediate with MI MI, and weakest with VI VI. Preference increased with overall terminal-link duration for all schedules, but the increase was more rapid for VI FI. The hyperbolic value-added model (HVA) and cumulative decision model (CDM) were fitted to the choice data and accounted for 83.9% and 76.1% of the variance, respectively. However, deviations from both models' predictions were systematic. A modification of the CDM, which assumed that reinforcers delivered after variable delays were more effective, improved the fit of the model and substantially reduced the systematic deviations in residuals. Comparable modifications of HVA produced only limited improvement. Results show that preference in concurrent chains with homogeneous terminal links depends on the degree of variability in reinforcer delays, and that the CDM can provide an excellent account of results across different terminal-link schedules.

Free operant observing in humans: a translational approach to compulsive certainty seeking

Excessive checking is reported in non-clinical populations and is a pervasive symptom in obsessive compulsive disorder (OCD). We implemented a free-operant task in humans, previously used in rats, wherein participants can "check" to reduce uncertainty. Participants can press an observing key to ascertain which of two main keys will, if pressed, currently lead to rewards. Over a series of experiments, we found that punishment robustly increased observing in non-clinical participants and that observing persisted long after punishment was removed. Moreover, participants appeared insensitive to the initial costs of checking, and a threefold increase in the effort required to observe served to deter participants only to a limited degree. We also assessed observing in OCD patients with no known comorbidities. The patients observed more than control participants and were abnormally insensitive to the introduction of punishment. These findings support the translational value of the task, with similar behaviours in humans and rodents. This paradigm may serve as a unifying platform, promoting interaction between different approaches to analyse adaptive and maladaptive certainty seeking behaviours. Specifically, we demonstrate how seemingly disparate theoretical and empirical approaches can be reconciled synergistically to promote a combined behavioural and cognitive account of certainty seeking.

Does a negative discriminative stimulus function as a punishing consequence?

The study and use of punishment in behavioral treatments has been constrained by ethical concerns. However, there remains a need to reduce harmful behavior unable to be reduced by differential-reinforcement procedures. We investigated whether response-contingent presentation of a negative discriminative stimulus previously correlated with an absence of reinforcers would punish behavior maintained by positive reinforcers. Across four conditions, pigeons were trained to discriminate between a positive discriminative stimulus (S+) signaling the presence of food, and a negative discriminative stimulus (S-) signaling the absence of food. Once learned, every five responses on average to the S+ produced S- for a duration of 1.5 s. S+ response rate decreased for a majority of pigeons when responses produced S-, compared to when they did not, or when a neutral control stimulus was presented. In Condition 5, choice between two concurrently presented S+ alternatives shifted away from the alternative producing S-, despite a 1:1 reinforcer ratio. Therefore, presenting contingent S- stimuli punishes operant behavior maintained on simple schedules and in choice situations. Development of negative discriminative stimuli as punishers of operant behavior could provide an effective approach to behavioral treatments for problem behavior and subverting suboptimal choices involved in addictions.

Control of saccadic latency in a dynamic environment: allocation of saccades in time follows the matching law

When exploring the visual environment, one uses saccades to shift gaze and fixation to gather spatially and temporally localized information. We propose that the temporal structure of our environment should constrain the temporal allocation of saccades. Here we probe the possibility of learning to control saccadic latencies in a choice paradigm. Six participants made saccades within 80–300 ms following a target horizontally stepping by 10° between two fixed locations. For each participant we constructed two classes of latencies, “short” and “long,” using the first and last quartiles of the individual baseline distribution (e.g., [80;152] ms and [185;300] ms, respectively). We then concurrently reinforced each class in three blocked conditions across ~60 experimental sessions per participant, using different reinforcement probabilities such that the relative ratio of reinforcement rates for short vs. long latencies was 9/1, 1/9, or 1/1. Latency distributions followed the reinforcement conditions: distributions shifted toward the shorter or longer values or became strongly bimodal. Moreover, the relative rates of short over long latencies matched the relative rates of reinforcers earned for the corresponding latencies (slope up to 0.95), which reveals the ability to choose when to saccade. Our results reveal that learned contingencies considerably affect the allocation of saccades in time and are in line with recent studies on the temporal adjustment of behavior to dynamic environments. This study provides strong evidence for fine operant control of saccadic latency, supporting the hypothesis of a cost-benefit control of saccade latencies.

NEW & NOTEWORTHY Saccades may be regarded as an information-foraging behavior mostly concerned with the spatial localization of objects, yet our world is dynamic and environmental temporal regularities should also affect saccade decisions. We present behavioral data from a choice task establishing that humans can learn to choose their saccadic latencies depending on the reinforcement contingencies. This suggests a cost-benefit-based policy that takes into account the learned temporal properties of the environmental contingencies for controlling saccade triggering.

Moving Beyond Reinforcement and Response Strength

Behavior analysis has often simultaneously depended upon and denied an implicit, hypothetical process of reinforcement as response strengthening. I discuss what I see as problematic about the use of such an implicit, possibly inaccurate, and likely unfalsifiable theory and describe issues to consider with respect to an alternative view without response strengthening. In my take on such an approach, important events (i.e., "reinforcers") provide a means to measure learning about predictive relations in the environment by modulating (i.e., inducing) performance dependent upon what is predicted and the relevant motivational mode or behavioral system active at that time (i.e., organismic state). Important events might be phylogenetically important, or they might acquire importance by being useful as signals for guiding an organism to where, when, or how currently relevant events might be obtained (or avoided). Given the role of learning predictive relations in such an approach, it is suggested that a potentially useful first step is to work toward formal descriptions of the structure of the predictive relations embodied in common facets of operant behavior (e.g., response-reinforcer contingencies, conditioned reinforcement, and stimulus control). Ultimately, the success of such an approach will depend upon how well it integrates formal characterizations of predictive relations (and how they are learned without response strengthening) and the relevant concomitant changes in organismic state across time. I also consider how thinking about the relevant processes in such a way might improve both our basic science and our technology of behavior.

Age differences in appetitive Pavlovian conditioning and extinction in rats

Mounting evidence indicates that adolescents exhibit heightened sensitivity to rewards and reward-related cues compared to adults, and that adolescents are often unable to exert behavioral control in the face of such cues. Moreover, differences in reward processing during adolescence have been linked to heightened risk taking and impulsivity. However, little is known about the processes by which adolescents learn about the appetitive properties of environmental stimuli that signal reward. To address this, Pavlovian conditioning procedures were used to test for differences in excitatory conditioning between adult and adolescent rats using various schedules of reinforcement. Specifically, separate cohorts of adult and adolescent rats were trained under conditions of consistent (continuous) or intermittent (partial) reinforcement. We found that the acquisition of anticipatory responding to a continuously-reinforced cue proceeded similarly in adolescents and adults. In contrast, responding increased at a greater rate in adolescents compared to adults during presentations of a partially-reinforced cue. We subsequently compared the ability of adolescent and adult rats to dynamically adjust the representation of a reward-predictive cue during extinction trials, in which a secondary inhibitory representation is acquired for the previously-reinforced stimulus. We observed significant age differences in the ability to flexibly update cue representations during extinction, in that the appetitive properties of cues with a history of either continuous or partial reinforcement persisted to a greater extent in adolescents relative to adults.

When good news leads to bad choices

Pigeons and other animals sometimes deviate from optimal choice behavior when given informative signals for delayed outcomes. For example, when pigeons are given a choice between an alternative that always leads to food after a delay and an alternative that leads to food only half of the time after a delay, preference changes dramatically depending on whether the stimuli during the delays are correlated with (signal) the outcomes or not. With signaled outcomes, pigeons show a much greater preference for the suboptimal alternative than with unsignaled outcomes. Key variables and research findings related to this phenomenon are reviewed, including the effects of durations of the choice and delay periods, probability of reinforcement, and gaps in the signal. We interpret the available evidence as reflecting a preference induced by signals for good news in a context of uncertainty. Other explanations are briefly summarized and compared.