Conditioned reinforcement: Experimental and theoretical issues

Evaluating functions of praise for children diagnosed with autism spectrum disorder

We assessed whether novel praise statements could be used to (a) maintain and increase responses with existing reinforcement histories and (b) teach a previously untaught response among children diagnosed with autism spectrum disorder across two experiments. During response-stimulus pairing, two responses resulted in preferred edibles but only one also produced a praise statement. In the absence of edibles, the response continuing to produce praise tended to persist more. Next, reversing the praise contingency tended to increase the other response. However, in no case did contingent delivery of those same praise statements result in the acquisition of untaught responses. These findings suggest that conditioning praise statements could serve different functions (antecedent or consequence) depending on the reinforcement history for particular responses.

The Use of Nonmonetary Outcomes in Health-Related Delay Discounting Research: Review and Recommendations

Delay discounting (DD) refers to the tendency to devalue an outcome as a function of its delay. Most contemporary human DD research uses hypothetical money to assess individual rates of DD. However, nonmonetary outcomes such as food, substances of misuse, and sexual outcomes have been used as well, and have advantages because of their connections to health. This article reviews the literature on the use of nonmonetary outcomes of food, drugs, and sexual outcomes in relation to health and reinforcer pathologies such as substance use disorders, obesity, and sexual risk behaviors, respectively, and makes a case for their use in discounting research. First, food, substances, and sex may be more ecologically valid outcomes than money in terms of their connections to health problems and reinforcer pathologies. Second, consistent trends in commodity-specific (i.e., domain) effects, in which nonmonetary outcomes are discounted more steeply than money, enhance variation in discounting values. Third, commodity-specific changes in discounting with treatments designed to change health choices are described. Finally, methodological trends such as test-retest reliability, magnitude effects, the use of hypothetical versus real outcomes, and age-related effects are discussed in relation to the three outcome types and compared to trends with monetary discounting. Limitations that center around individual preferences, nonsystematic data, and deprivation are discussed. We argue that researchers can enhance their DD research, especially those related to health problems and reinforcer pathologies, with the use of nonmonetary outcomes. Recommendations for future directions of research are delineated.

Rethinking Norm Psychology

Norms permeate human life. Most of people's activities can be characterized by rules about what is appropriate, allowed, required, or forbidden-rules that are crucial in making people hyper-cooperative animals. In this article, I examine the current cognitive-evolutionary account of "norm psychology" and propose an alternative that is better supported by evidence and better placed to promote interdisciplinary dialogue. The incumbent theory focuses on rules and claims that humans genetically inherit cognitive and motivational mechanisms specialized for processing these rules. The cultural-evolutionary alternative defines normativity in relation to behavior-compliance, enforcement, and commentary-and suggests that it depends on implicit and explicit processes. The implicit processes are genetically inherited and domain-general; rather than being specialized for normativity, they do many jobs in many species. The explicit processes are culturally inherited and domain-specific; they are constructed from mentalizing and reasoning by social interaction in childhood. The cultural-evolutionary, or "cognitive gadget," perspective suggests that people alive today-parents, educators, elders, politicians, lawyers-have more responsibility for sustaining normativity than the nativist view implies. People's actions not only shape and transmit the rules, but they also create in each new generation mental processes that can grasp the rules and put them into action.

Effects of inter-trial interval on sign-tracking and conditioned reinforcer efficacy in female rats

Previous nonhuman studies have reported that sign-tracking to a conditioned stimulus (CS) is increased when the intertrial interval (ITI) duration is increased. Separate studies indicate that individual differences in sign-tracking (vs. goal-tracking) at a fixed ITI (and CS duration) is predictive of the conditioned reinforcer efficacy of the CS. The present study evaluates, for the first time, if increasing the ITI increases rats' sign-tracking and the conditioned reinforcing efficacy of the CS. Forty-five female rats were randomly assigned to one of three groups that completed appetitive Pavlovian training with ITIs of 14, 24, or 96 s. Subsequently, they completed tests of conditioned reinforcement. Replicating previous findings, longer ITIs increased sign-tracking to a lever-CS and, extending the literature, conditioned reinforcer efficacy of that CS was highest at the longest ITI used during Pavlovian training. Implications for behavioral interventions using conditioned reinforcement are discussed.

Pavlovian learning and conditioned reinforcement

Conditioned reinforcers are widely used in applied behavior analysis. Basic research evidence reveals that Pavlovian learning plays an important role in the acquisition and efficacy of new conditioned-reinforcer functions. Thus, a better understanding of Pavlovian principles holds the promise of improving the efficacy of conditioned reinforcement in applied research and practice. This paper surveys how (and if) Pavlovian principles are presented in behavior-analytic textbooks; imprecisions and knowledge gaps within contemporary Pavlovian empirical findings are highlighted. Thereafter, six practical principles of Pavlovian conditioning are presented along with empirical support and knowledge gaps that should be filled by applied and translational behavior-analytic researchers. Innovative applications of these principles are outlined for research in language acquisition, token reinforcement, and self-control.

Models of Conditioned Reinforcement and Abnormal Behaviour in Captive Animals

Abnormal behaviours are common in captive animals, and despite a lot of research, the development, maintenance and alleviation of these behaviours are not fully understood. Here, we suggest that conditioned reinforcement can induce sequential dependencies in behaviour that are difficult to infer from direct observation. We develop this hypothesis using recent models of associative learning that include conditioned reinforcement and inborn facets of behaviour, such as predisposed responses and motivational systems. We explore three scenarios in which abnormal behaviour emerges from a combination of associative learning and a mismatch between the captive environment and inborn predispositions. The first model considers how abnormal behaviours, such as locomotor stereotypies, may arise from certain spatial locations acquiring conditioned reinforcement value. The second model shows that conditioned reinforcement can give rise to abnormal behaviour in response to stimuli that regularly precede food or other reinforcers. The third model shows that abnormal behaviour can result from motivational systems being adapted to natural environments that have different temporal structures than the captive environment. We conclude that models including conditioned reinforcement offer an important theoretical insight regarding the complex relationships between captive environments, inborn predispositions, and learning. In the future, this general framework could allow us to further understand and possibly alleviate abnormal behaviours.

Topographical aspects of brief-stimulus presentations: A re-examination of the problem of conditioned reinforcement

Three pigeons were exposed to second-order schedules in which responding under a fixed-interval (FI) component schedule was reinforced according to a variable-interval (VI) schedule of food reinforcement. Completion of each component resulted in either (1) brief presentation of a stimulus present during reinforcement (paired brief stimulus), (2) brief presentation of a stimulus not present during reinforcement (nonpaired brief stimulus), or (3) no stimulus presentation (tandem schedule). Under the two nonpaired brief stimulus conditions, either a change in keylight color or onset of houselight illumination was used as the brief stimulus. Similar patterns of keypecking occurred under tandem and nonpaired keylight brief-stimulus presentations, whereas nonpaired houselight brief-stimulus presentations generated positively accelerated within-component keypeck patterning for two pigeons. When the same keylight brief stimulus was paired with food, positively accelerated patterns of keypecking were obtained for all pigeons. Differences in the effects of nonpaired brief-stimulus presentations on second-order schedule performance suggest that component schedule patterning under nonpaired brief-stimulus procedures is a function of the particular type of stimulus used (i.e., houselight versus keylight). These results suggest that (1) brief houselight illumination may function as a sensory reinforcer, and (2) a briefly presented food-paired stimulus can function as an effective conditioned reinforcer.

An evaluation of methods for studying the effects of conditioned reinforcement on human choice

Shahan et al. (2006) found that the relative rate of pigeons' pecking on two observing responses (i.e., responses that only produced an S+ or stimulus correlated with primary reinforcement) was well described by the relative rate of S+ delivery. Researchers have not evaluated the effects of S+ delivery rate in a concurrent observing response procedure with human subjects, so the necessary procedural modifications for studying the effects of conditioned reinforcement on human choice remain unclear. The purpose of the current study was to conduct an additive component analysis of modifications to the procedures of Shahan et al. (2006). We evaluated the additive effects of introducing response cost, a changeover response, and ordinal discriminative stimuli on correspondence with the results of Shahan et al. and the quality of fits of the generalized matching equation. When our procedures were most similar to those of Shahan et al., we observed low rates of observing and indifference between the two observing responses. For the group of subjects with whom all three additive components were included, we obtained the highest level of sensitivity to relative rate of S+ delivery, but the slope and R² of our fits of the generalized matching equation were still much lower than those obtained by Shahan et al. Potential reasons for these discrepancies, methods of resolving them, and implications for future research are discussed.

Applied behavior analysis and the zoo: Forthman and Ogden (1992) thirty years later

The field of applied behavior analysis has been directly involved in both research and applications of behavioral principles to improve the lives of captive zoo animals. Thirty years ago, Forthman and Ogden (1992) wrote one of the first papers documenting some of these efforts. Since that time, considerable work has been done using behavioral principles and procedures to guide zoo welfare efforts. The current paper reexamines and updates Forthman and Ogden's original points, with attention to the 5 categories they detailed: (a) promotion of species-typical behavior, (b) reintroduction and repatriation of endangered species, (c) animal handling, (d) pest control, and (e) animal performances. In addition, we outline 3 current and future directions for behavior analytic endeavors: (a) experimental analyses of behavior and the zoo, (b) applied behavior analysis and the zoo, and (c) single-case designs and the zoo. The goal is to provide a framework that can guide future behavioral research in zoos, as well as create applications based on these empirical evaluations.

Grounding applied animal behavior practices in the experimental analysis of behavior

Some of the earliest applications outside the laboratory of principles derived from the experimental analysis of behavior (EAB), such as the pioneering work of Keller and Marian Breland, involved animals. This translational tradition continues to the present as EAB-related behavior principles are applied with increasing frequency to behavior management and training practices with animals in nonlaboratory settings. Such translations, and those populations to which they are applied, benefit from a rigorous experimental analysis of practices that are promulgated in popular outlets. These translations both affirm the generality of those principles and serve as goads for laboratory and field research that can further articulate extant principles, develop new ones, and refine methods of application and assessment. This review considered several areas of basic EAB research and contemporary applied animal behavior (AAB) practices in relation to one another: (1) response establishment and maintenance, (2) response reduction and elimination, (3) chaining and conditioned reinforcement, and (4) discriminative stimulus control. Within each topic, a selection of processes and procedures in both EAB and AAB work were reviewed in relation to one another.

Food cue reactivity: Neurobiological and behavioral underpinnings

The modern obesogenic environment contains an abundance of food cues (e.g., sight, smell of food) as well cues that are associated with food through learning and memory processes. Food cue exposure can lead to food seeking and excessive consumption in otherwise food-sated individuals, and a high level of food cue responsivity is a risk factor for overweight and obesity. Similar food cue responses are observed in experimental rodent models, and these models are therefore useful for mechanistically identifying the neural circuits mediating food cue responsivity. This review draws from both experimental rodent models and human data to characterize the behavioral and biological processes through which food-associated stimuli contribute to overeating and weight gain. Two rodent models are emphasized - cue-potentiated feeding and Pavlovian-instrumental transfer - that provide insight in the neural circuits and peptide systems underlying food cue responsivity. Data from humans are highlighted that reveal physiological, psychological, and neural mechanisms that connect food cue responsivity with overeating and weight gain. The collective literature identifies connections between heightened food cue responsivity and obesity in both rodents and humans, and identifies underlying brain regions (nucleus accumbens, amygdala, orbitofrontal cortex, hippocampus) and endocrine systems (ghrelin) that regulate food cue responsivity in both species. These species similarities are encouraging for the possibility of mechanistic rodent model research and further human research leading to novel treatments for excessive food cue responsivity in humans.

Modelling how cleaner fish approach an ephemeral reward task demonstrates a role for ecologically tuned chunking in the evolution of advanced cognition

What makes cognition “advanced” is an open and not precisely defined question. One perspective involves increasing the complexity of associative learning, from conditioning to learning sequences of events (“chaining”) to representing various cue combinations as “chunks.” Here we develop a weighted graph model to study the mechanism enabling chunking ability and the conditions for its evolution and success, based on the ecology of the cleaner fish Labroides dimidiatus. In some environments, cleaners must learn to serve visitor clients before resident clients, because a visitor leaves if not attended while a resident waits for service. This challenge has been captured in various versions of the ephemeral reward task, which has been proven difficult for a range of cognitively capable species. We show that chaining is the minimal requirement for solving this task in its common simplified laboratory format that involves repeated simultaneous exposure to an ephemeral and permanent food source. Adding ephemeral–ephemeral and permanent–permanent combinations, as cleaners face in the wild, requires individuals to have chunking abilities to solve the task. Importantly, chunking parameters need to be calibrated to ecological conditions in order to produce adaptive decisions. Thus, it is the fine-tuning of this ability, which may be the major target of selection during the evolution of advanced associative learning.

What makes cognition ‘advanced’ is an open and not precisely defined question. In this study, a cognitive model of cleaner fish learning the ephemeral-reward task demonstrates how a critical step in cognitive evolution may be understood as the evolution of chunking and its tuning to fit ecological conditions.

Associative processes in addiction relapse models: A review of their Pavlovian and instrumental mechanisms, history, and terminology

Animal models of relapse to drug-seeking have borrowed heavily from associative learning approaches. In studies of relapse-like behaviour, animals learn to self-administer drugs then receive a period of extinction during which they learn to inhibit the operant response. Several triggers can produce a recovery of responding which form the basis of a variety of models. These include the passage of time (spontaneous recovery), drug availability (rapid reacquisition), extinction of an alternative response (resurgence), context change (renewal), drug priming, stress, and cues (reinstatement). In most cases, the behavioural processes driving extinction and recovery in operant drug self-administration studies are similar to those in the Pavlovian and behavioural literature, such as context effects. However, reinstatement in addiction studies have several differences with Pavlovian reinstatement, which have emerged over several decades, in experimental procedures, associative mechanisms, and terminology. Interestingly, in cue-induced reinstatement, drug-paired cues that are present during acquisition are omitted during lever extinction. The unextinguished drug-paired cue may limit the model’s translational relevance to cue exposure therapy and renders its underlying associative mechanisms ambiguous. We review major behavioural theories that explain recovery phenomena, with a particular focus on cue-induced reinstatement because it is a widely used model in addiction. We argue that cue-induced reinstatement may be explained by a combination of behavioural processes, including reacquisition of conditioned reinforcement and Pavlovian to Instrumental Transfer. While there are important differences between addiction studies and the behavioural literature in terminology and procedures, it is clear that understanding associative learning processes is essential for studying relapse.

The click is not the trick: the efficacy of clickers and other reinforcement methods in training naïve dogs to perform new tasks

Background

A handheld metal noisemaker known as a “clicker” is widely used to train new behaviors in dogs; however, evidence for their superior efficacy compared to providing solely primary reinforcement or other secondary reinforcers in the acquisition of novel behavior in dogs is largely anecdotal.

Methods

Three experiments were conducted to determine under what circumstances a clicker secondary reinforcer may result in acquisition of a novel behavior more rapidly or to a higher level compared to other readily available reinforcement methods. In Experiment 1, three groups of 30 dogs each were shaped to emit a novel sit and stay behavior of increasing duration with either the delivery of food alone, a verbal stimulus paired with food, or a clicker with food. The group that received only a primary reinforcer reached a significantly higher criterion of training success than the group trained with a verbal secondary reinforcer. Performance of the group experiencing a clicker as a secondary reinforcer was intermediate between the other two groups, but not significantly different from either. In Experiment 2, three groups of 25 dogs each were shaped to emit a nose targeting behavior and then perform that behavior at increasing distances from the experimenter using the same three methods of positive reinforcement as in Experiment 1. No statistically significant differences between the groups were found. In Experiment 3, three groups of 30 dogs each were shaped to emit a nose-targeting behavior upon an array of wooden blocks with task difficulty increasing throughout testing using the same three methods of positive reinforcement as previously tested. No statistically significant differences between the groups were found.

Results

Overall, the findings suggest that both primary reinforcement alone as well as a verbal or clicker secondary reinforcer can be used successfully in training a dog to perform a novel behavior, but that no positive reinforcement method demonstrated significantly greater efficacy than any other.

Big Surprises: Jackpot Reinforcers in Research and Practice

Unusually large, infrequent reinforcers, described as jackpots, are the subject of considerable discussion among applied animal behaviorists. Such reinforcers offer considerable promise in applied behavior analysis as a means of both potentiating training of new responses and response classes and enhancing previously learned ones. The concept of jackpot reinforcement, however, is rife with not only a lack of definitional and procedural clarity but also a paucity of research, either basic or applied, on such reinforcement. Considerations in undertaking such research include defining the parameters of jackpot reinforcers, identifying suitable dependent variables, and creating experimental designs appropriate for their assessment. The few experimental analyses of jackpot reinforcer effects on either response acquisition or maintenance have produced little evidence of systematic effects, despite the use of several different methods and behavioral measures. Negative results could reflect either the absence of systematic effects of these jackpot reinforcers or unsuitable methods of analysis. Only further research will take the topic of jackpot reinforcers beyond opinion and testimonial and into the realm of the science of behavior.

The pharmacological stressor yohimbine, but not U50,488, increases responding for conditioned reinforcers paired with ethanol or sucrose

RATIONALE

Environmental stimuli paired with alcohol can function as conditioned reinforcers (CRfs) and trigger relapse to alcohol-seeking. In animal models, pharmacological stressors can enhance alcohol consumption and reinstate alcohol-seeking, but it is unknown whether stress can potentiate the conditioned reinforcing properties of alcohol-paired stimuli.

OBJECTIVES

We examined whether the pharmacological stressors, the α-2 adrenoreceptor antagonist yohimbine (vehicle, 1.25, 2.5 mg/kg; IP) and the K-opioid receptor agonist U50,488 (vehicle, 1.25, 2.5 mg/kg; SC), increase responding for conditioned reinforcement, and if their effects interact with the nature of the reward (alcohol vs. sucrose). We subsequently examined the effects of yohimbine (vehicle, 1.25, 2.5 mg/kg; IP) on responding for sensory reinforcement.

METHODS

Male Long-Evans underwent Pavlovian conditioning, wherein a tone-light conditioned stimulus (CS) was repeatedly paired with 12% EtOH or 21.7% sucrose. Next, tests of responding for a CRf were conducted. Responding on the CRf lever delivered the CS, whereas responding on the other lever had no consequences. In a separate cohort of rats, the effects of yohimbine on responding just for the sensory reinforcer were examined.

RESULTS

Both doses of yohimbine, but not U50,488, increased responding for conditioned reinforcement. This enhancement occurred independently of the nature of the reward used during Pavlovian conditioning. Yohimbine-enhanced responding for a CRf was reproducible and stable over five tests; it even persisted when the CS was omitted. Both doses of yohimbine also increased responding for sensory reinforcement.

CONCLUSIONS

Yohimbine increases operant responding for a variety of sensory and conditioned reinforcers. This enhancement may be independent of its stress-like effects.

Effects of dose on acquisition and persistence of a new response for a remifentanil-associated stimulus

Previous research demonstrated that a remifentanil-associated stimulus facilitated the acquisition of a previously unlearned response; however, it is unclear how long a remifentanil-associated stimulus maintains conditioned reinforcing properties under conditions of daily testing. To address this gap, we exposed adult male rats to response-independent stimulus presentations and deliveries of remifentanil (1.0, 3.2, or 10.0 μg/kg/infusion). Rats either received the stimulus presentations and remifentanil deliveries together (Paired Pavlovian conditioning) or according to separate clocks (Random control group). In the sessions following Pavlovian conditioning, we allowed rats to emit nose-poke responses for the presentation of the stimulus alone and measured the extent to which the stimulus facilitated and maintained a previously unlearned response. We tested responding for the stimulus presentations across 28 daily sessions to assess the Pavlovian extinction (degradation of the drug-stimulus association) of the conditioned reinforcing properties of the remifentanil-associated stimulus. We observed the highest and most persistent levels of responding in rats with a Paired Pavlovian conditioning history at 3.2 and 10.0 μg/kg/infusion. In addition, we included analyses of the variability in responding for each group, which revealed individual differences in the susceptibility of the remifentanil-associated stimulus acting as a conditioned reinforcer. These findings demonstrate that a remifentanil-associated stimulus has the ability to sustain drug-seeking behavior and underscores the importance of Pavlovian conditioning in promoting drug abuse.

What's in a Click? The Efficacy of Conditioned Reinforcement in Applied Animal Training: A Systematic Review and Meta-Analysis

Simple Summary

Conditioned reinforcement, for example, clicker training, has become increasingly popular in recent decades. Hence, questions about the effectiveness of the conditioned reinforcer have become prominent in the animal training arena. This article summarizes the scientific literature on conditioned reinforcement in applied animal training settings (e.g., homes). It was found that dogs and horses were the most frequently studied animals. Clickers and food were the most often used training stimuli. Effect size analysis found a medium effect of clicker training. The literature reviewed here shows that conditioned reinforcement is an effective approach to change animal behavior; however, sizable information potentially related to its effectiveness was not clearly reported in the studies (e.g., food preferences). Although this review fills in a gap in the literature, it also points to the need for more research to further the understanding of conditioned reinforcement phenomena.

Abstract

A conditioned reinforcer is a stimulus that acquired its effectiveness to increase and maintain a target behavior on the basis of the individual’s history—e.g., pairings with other reinforcers. This systematic review synthesized findings on conditioned reinforcement in the applied animal training field. Thirty-four studies were included in the review and six studies were eligible for a meta-analysis on the effectiveness of behavioral interventions that implemented conditioned reinforcement (e.g., clicks, spoken word, or whistles paired with food). The majority of studies investigated conditioned reinforcement with dogs (47%, n = 16) and horses (30%, n = 10) implementing click–food pairings. All other species (cats, cattle, fish, goats, and monkeys) were equally distributed across types of conditioned (e.g., clicker or spoken word) and unconditioned reinforcers (e.g., food, water, or tactile). A meta-analysis on the effectiveness of conditioned reinforcement in behavioral interventions found a medium summary effect size (Tau-U 0.77; CI_95% = [0.53, 0.89]), when comparing baseline, where no training was done, and treatment levels. Moderators of conditioned reinforcement effectiveness were species (e.g., horses) and research design (e.g., multiple-baseline designs). The small number of intervention-focused studies available limits the present findings and highlights the need for more systematic research into the effectiveness of conditioned reinforcement across species.

Learning steers the ontogeny of an efficient hunting sequence in zebrafish larvae

Goal-directed behaviors may be poorly coordinated in young animals but, with age and experience, behavior progressively adapts to efficiently exploit the animal’s ecological niche. How experience impinges on the developing neural circuits of behavior is an open question. We have conducted a detailed study of the effects of experience on the ontogeny of hunting behavior in larval zebrafish. We report that larvae with prior experience of live prey consume considerably more prey than naive larvae. This is mainly due to increased capture success and a modest increase in hunt rate. We demonstrate that the initial turn to prey and the final capture manoeuvre of the hunting sequence were jointly modified by experience and that modification of these components predicted capture success. Our findings establish an ethologically relevant paradigm in zebrafish for studying how the brain is shaped by experience to drive the ontogeny of efficient behavior.

A-learning: A new formulation of associative learning theory

We present a new mathematical formulation of associative learning focused on non-human animals, which we call A-learning. Building on current animal learning theory and machine learning, A-learning is composed of two learning equations, one for stimulus-response values and one for stimulus values (conditioned reinforcement). A third equation implements decision-making by mapping stimulus-response values to response probabilities. We show that A-learning can reproduce the main features of: instrumental acquisition, including the effects of signaled and unsignaled non-contingent reinforcement; Pavlovian acquisition, including higher-order conditioning, omission training, autoshaping, and differences in form between conditioned and unconditioned responses; acquisition of avoidance responses; acquisition and extinction of instrumental chains and Pavlovian higher-order conditioning; Pavlovian-to-instrumental transfer; Pavlovian and instrumental outcome revaluation effects, including insight into why these effects vary greatly with training procedures and with the proximity of a response to the reinforcer. We discuss the differences between current theory and A-learning, such as its lack of stimulus-stimulus and response-stimulus associations, and compare A-learning with other temporal-difference models from machine learning, such as Q-learning, SARSA, and the actor-critic model. We conclude that A-learning may offer a more convenient view of associative learning than current mathematical models, and point out areas that need further development.

Reducing impulsive choice VII: effects of duration of delay-exposure training

Impulsive choice is related to substance use disorders, obesity, and other behaviors that negatively impact human health. Reducing impulsive choice may prove beneficial in ameliorating these maladaptive behaviors. Preclinical research in rats indicates that one reliable method for producing large and lasting reductions in impulsive choice is delay-exposure (DE) training. In all six of the prior DE-training experiments, rats were given extensive experience (~ 120 training sessions) with a delayed reinforcement contingency. The present experiment evaluated if similar large and lasting reductions in impulsive choice could be achieved with less training. The duration of DE training between groups of male Wistar rats was 0 sessions (training ended after a lever-pressing acquisition criterion was met), 30, 60, or 120 sessions. Comparison groups were given the same durations of training with immediate reinforcement. A post-training assessment of impulsive choice was completed using an increasing-delay procedure. For rats assigned to the 60-session condition, impulsive choice was reassessed at a 120-day follow-up. DE training reduced impulsive choice but, contrary to expectation, reductions in impulsive choice did not increase with DE-training duration (no significant training-duration by group interaction). Importantly, 60 sessions of DE training produced reductions in impulsive choice that were comparable to prior published findings and this effect remained significant at the 120-day follow-up. Procedural elements that may be responsible for the DE-training effect, and how they could be improved in future experiments, are discussed.

Appetitive traits as targets for weight loss: The role of food cue responsiveness and satiety responsiveness

Individuals with overweight or obesity (OW/OB) are at increased risk for significant physical and psychological comorbidities. The current treatment for OW/OB is behavioral weight loss, which provides psychoeducation on nutrition and physical activity, as well as behavior therapy skills. However, behavioral weight loss is not effective for the majority of the individuals who participate. Research suggests that overeating, or eating past nutritional needs, is one of the leading causes of weight gain. Accumulating evidence suggests that appetitive traits, such as food cue responsiveness and satiety responsiveness, are associated with overeating and weight in youth and adults. The following review presents the current literature on the relationship between food cue responsiveness, satiety responsiveness, overeating, and OW/OB. Research suggests that higher food cue responsiveness and lower satiety responsiveness are associated with overeating and OW/OB cross-sectionally and longitudinally. Emerging data suggest that food cue responsiveness and satiety responsiveness may exist along the same continuum and can be targeted to manage overeating and reduce weight. We have developed a treatment model targeting food cue responsiveness and satiety responsiveness to reduce overeating and weight and have preliminary feasibility, acceptability, and efficacy data, with testing currently being conducted in larger trials. Through programs targeting appetitive traits we hope to develop an alternative weight loss model to assist individuals with a propensity to overeat.

Ethanol-paired stimuli can increase reinforced ethanol responding

While ethanol-paired stimuli are frequently postulated to increase drinking motivation and thus increase ethanol responding and precipitate relapse, no study has demonstrated increases in ethanol-reinforced responding following presentation of an ethanol-paired stimulus that had not previously been part of a contingent relationship. Previous studies have shown that food-paired stimuli can increase food responding that is at low rates and increase food consumption in food-sated rats. In Experiment 1, we show that an ethanol-paired stimulus can increase ethanol responding that is at low levels late in the experimental session, presumably due to satiation. However, these increases may have resulted from either associative or non-associative mechanisms. In Experiment 2, we compared the effects of an ethanol-paired stimulus to those of the same stimulus in a Truly-Random-Control group. In a Truly-Random-Control, the stimulus and ethanol each are presented on independent random schedules, and thus any differences between the effects of the stimulus in the experimental and control groups is likely attributable to the association between the stimulus and ethanol. The stimulus increased ethanol-reinforced responding in both the experimental and control groups, but these increases were greater in the experimental than the control group. Thus, both stimulus-change and the pairing of the stimulus with ethanol may result in increases in ethanol-reinforced responding.

Clicker Training Accelerates Learning of Complex Behaviors but Reduces Discriminative Abilities of Yucatan Miniature Pigs

Simple Summary

Animal training is intended to teach specific behavioral responses to specific requests. Clicker Training (CT) is a method to train animals based on the use of a device that emits a sound to be associated as a marker that predicts the delivery of something wanted (food). It is believed that CT decreases training time compared to other types of training that use different markers, such as voice. Herein, we used two-month-old miniature piglets to assess whether CT decreased the number of repeats required to learn complex behaviors compared to voice-trained animals. Furthermore, we compared the number of correct choices of animals from both groups when tested for the discrimination of objects. The results indicated that CT decreased the number of repetitions required to learn to fetch an object but reduced the animals’ ability to make correct decisions during discriminatory trials compared to voice-trained animals. This suggests that CT is more efficient than voice in teaching complex behaviors but reduces the ability of animals to use the cognitive processes necessary to discriminate and select objects associated with reward. Animal trainers might consider our results to decide which marker is to be implemented based on the aim and purpose of the training.

Abstract

Animal training is meant to teach specific behavioral responses to specific cues. Clicker training (CT) is a popular training method based on the use of a device that emits a sound of double-click to be associated as a first-order conditioned stimulus in contingency with positive reinforcements. After some repetitions, the clicker sound gains some incentive value and can be paired with the desired behavior. Animal trainers believed that CT can decrease training time compared to other types of training. Herein, we used two-month old miniature piglets to evaluate whether CT decreased the number of repetitions required to learn complex behaviors as compared with animals trained with voice instead of the clicker. In addition, we compared the number of correct choices of animals from both groups when exposed to object discriminative tests. Results indicated that CT decreased the number of repetitions required for pigs to learn to fetch an object but reduced the ability of animals to make correct choices during the discriminate trials. This suggests that CT is more efficient than voice to teach complex behaviors but reduces the ability of animals to use cognitive processes required to discriminate and select objects associated with reward.

The Near-Miss Effect in Slot Machines: A Review and Experimental Analysis Over Half a Century Later

In games of chance, a near miss is said to occur when feedback for a loss approximates a win. For instance, obtaining “cherry–cherry–lemon” on a slot machine could be considered a near miss. Sixty-six years ago, B.F. Skinner first proposed the idea that near-miss events might reinforce continued play in slot machines, and despite some inconsistencies in the experimental literature, belief in this “near-miss effect” has remained strong. In the present manuscript, we will review this literature and present experimental assessments of the near-miss effect on the frequency of the gambling response. Experiment 1 used a tightly controlled resistance-to-extinction procedure in pigeons to evaluate the putative reinforcing effect of near misses relative to a control “far-miss” reel pattern. Experiment 2 extended Experiment 1’s procedure to human participants. The results of both experiments failed to support the near-miss effect hypothesis. Experiment 3 used a further simplified procedure to assess the validity of the resistance-to-extinction paradigm when a probable conditional reinforcer was present on the reel stimuli. Although a clear conditional response was obtained from the reel, subsequent testing in extinction revealed no conditionally reinforcing function of this stimulus on operant response frequency.

Effects of rat strain and method of inducing ethanol drinking on Pavlovian-Instrumental-Transfer with ethanol-paired conditioned stimuli

Ethanol-paired conditioned stimuli (CSs) are widely thought to invigorate ethanol responding, and thus, precipitate relapse to drinking. However, preclinical studies investigating this issue using Pavlovian-Instrumental-Transfer (PIT) procedures have had mixed results, with some studies finding PIT while others did not. The studies failing to show PIT used Lewis rats and induced ethanol drinking using a post-prandial drinking procedure. The present experiments examined whether either of these two variables influenced the magnitude of PIT observed. In the first experiment, ethanol drinking in Lewis rats was induced using either sucrose fading or post-prandial drinking. In the second experiment, ethanol drinking was induced using post-prandial drinking in either Long-Evans Hooded or Lewis rats. In both experiments, rats were trained to respond for ethanol under a random interval schedule. Subsequently with the lever removed, 2-min light presentations were paired with ethanol deliveries. Finally, with the lever returned, the effect of light presentations on responding was tested while responding was in extinction. Light presentations similarly affected responding in Lewis rats regardless of the method of drinking induction. Likewise, light presentations similarly affected responding in both Lewis and Long-Evans Hooded rats. Neither ethanol induction method nor rat strain affected the magnitude of PIT observed, and thus, neither likely explains previous failures to observe PIT with ethanol-maintained behavior.

Social learning through associative processes: a computational theory

Social transmission of information is a key phenomenon in the evolution of behaviour and in the establishment of traditions and culture. The diversity of social learning phenomena has engendered a diverse terminology and numerous ideas about underlying learning mechanisms, at the same time that some researchers have called for a unitary analysis of social learning in terms of associative processes. Leveraging previous attempts and a recent computational formulation of associative learning, we analyse the following learning scenarios in some generality: learning responses to social stimuli, including learning to imitate; learning responses to non-social stimuli; learning sequences of actions; learning to avoid danger. We conceptualize social learning as situations in which stimuli that arise from other individuals have an important role in learning. This role is supported by genetic predispositions that either cause responses to social stimuli or enable social stimuli to reinforce specific responses. Simulations were performed using a new learning simulator program. The simulator is publicly available and can be used for further theoretical investigations and to guide empirical research of learning and behaviour. Our explorations show that, when guided by genetic predispositions, associative processes can give rise to a wide variety of social learning phenomena, such as stimulus and local enhancement, contextual imitation and simple production imitation, observational conditioning, and social and response facilitation. In addition, we clarify how associative mechanisms can result in transfer of information and behaviour from experienced to naive individuals.

What can associative learning do for planning?

There is a new associative learning paradox. The power of associative learning for producing flexible behaviour in non-human animals is downplayed or ignored by researchers in animal cognition, whereas artificial intelligence research shows that associative learning models can beat humans in chess. One phenomenon in which associative learning often is ruled out as an explanation for animal behaviour is flexible planning. However, planning studies have been criticized and questions have been raised regarding both methodological validity and interpretations of results. Due to the power of associative learning and the uncertainty of what causes planning behaviour in non-human animals, I explored what associative learning can do for planning. A previously published sequence learning model which combines Pavlovian and instrumental conditioning was used to simulate two planning studies, namely Mulcahy & Call 2006 'Apes save tools for future use.' Science 312, 1038-1040 and Kabadayi & Osvath 2017 'Ravens parallel great apes in flexible planning for tool-use and bartering.' Science 357, 202-204. Simulations show that behaviour matching current definitions of flexible planning can emerge through associative learning. Through conditioned reinforcement, the learning model gives rise to planning behaviour by learning that a behaviour towards a current stimulus will produce high value food at a later stage; it can make decisions about future states not within current sensory scope. The simulations tracked key patterns both between and within studies. It is concluded that one cannot rule out that these studies of flexible planning in apes and corvids can be completely accounted for by associative learning. Future empirical studies of flexible planning in non-human animals can benefit from theoretical developments within artificial intelligence and animal learning.

Function matters: a review of terminological differences in applied and basic clicker training research

In clicker training, animal trainers pair a small device (a “clicker”) with a reward when teaching or maintaining responding. Animal trainers often assume clicker training is a “science-based” way to train animals. But, the few studies that have compared clicker training to a control have not provided evidence that adding a clicker is beneficial to training. This may be because research on clicker training has studied only one of several potential functions of the clicker stimulus that have been discussed by animal trainers. A systematic approach to researching the function of the clicker in clicker training would benefit from collaboration between applied and basic researchers. However, this will require that terminological differences between animal trainers and basic researchers are reconciled. This paper reviews the few studies that have compared clicker training to a control group and then discusses how trainers and basic researchers use the same terminology in functionally different ways—suggesting the empirical support for mechanisms underlying clicker training is less robust than previously assumed. These differences highlight many opportunities to answer basic and applied research questions relative to clicker training methods. Advancements in clicker training methods will benefit animal trainers who have been using clicker training for decades as well as applied practitioners who have extended clicker training to humans in educational and clinical settings.

Stimuli previously associated with reinforcement mitigate resurgence

Resurgence refers to the recurrence of an extinguished target behavior following subsequent suspension of alternative reinforcement. Delivery of reinforcers during extinction of alternative behavior has been shown to mitigate resurgence. The present experiment aimed to determine whether delivering stimuli associated with reinforcers during resurgence testing similarly mitigates resurgence. Three groups of rats pressed target levers for food according to variable-interval 15-s schedules during Phase 1. In Phase 2, lever pressing was extinguished, and an alternative nose-poke response produced alternative reinforcement according to a variable-interval 15-s schedule. Food reinforcement was always associated with illumination of the food aperture and an audible click from the pellet dispenser during Phases 1 and 2. Phase 3 treatments differed between groups. For one group, nose poking continued to produce food and food-correlated stimuli. Both of these consequences were suspended for a second group. Finally, nose poking produced food-correlated stimuli but not food for a third group. Target-lever pressing resurged in the group that received no consequences and in the group that received only food-correlated stimuli for nose poking. Resurgence, however, was smaller for the group that received food-correlated stimuli than for the group that received no consequences for nose poking. Target-lever pressing did not increase between phases in the group that continued to receive food and associated stimuli. Thus, delivery of stimuli associated with food reinforcement after suspension of food reduced but did not eliminate resurgence of extinguished lever pressing. These findings contribute to potential methodologies for preventing relapse of extinguished problem behavior in clinical settings.

Suboptimal choice in rats: Incentive salience attribution promotes maladaptive decision-making

Stimuli that are more predictive of subsequent reward also function as better conditioned reinforcers. Moreover, stimuli attributed with incentive salience function as more robust conditioned reinforcers. Some theories have suggested that conditioned reinforcement plays an important role in promoting suboptimal choice behavior, like gambling. The present experiments examined how different stimuli, those attributed with incentive salience versus those without, can function in tandem with stimulus-reward predictive utility to promote maladaptive decision-making in rats. One group of rats had lights associated with goal-tracking as the reward-predictive stimuli and another had levers associated with sign-tracking as the reward-predictive stimuli. All rats were first trained on a choice procedure in which the expected value across both alternatives was equivalent but differed in their stimulus-reward predictive utility. Next, the expected value across both alternatives was systematically changed so that the alternative with greater stimulus-reward predictive utility was suboptimal in regard to primary reinforcement. The results demonstrate that in order to obtain suboptimal choice behavior, incentive salience alongside strong stimulus-reward predictive utility may be necessary; thus, maladaptive decision-making can be driven more by the value attributed to stimuli imbued with incentive salience that reliably predict a reward rather than the reward itself.

Effects of sex and remifentanil dose on rats' acquisition of responding for a remifentanil-conditioned reinforcer

Opioid-conditioned reinforcement is thought to exacerbate opioid abuse and dependence. Sex/gender can influence opioid abuse behaviors, but the effects of sex/gender on opioid-conditioned reinforcement, specifically, are unclear. In this study, we compared new-response acquisition with opioid-conditioned reinforcement in male and female rats. First, separate groups received response-independent remifentanil injections (0.0-32.0 μg/kg, intravenous) and presentations of a light-noise stimulus. In the experimental groups, injections and stimulus presentations always co-occurred [paired Pavlovian conditioning (PAV)]; in the control groups, the two occurred independently of each other (random PAV). Next, in the instrumental acquisition (ACQ) sessions, two novel nose-poke manipulanda were introduced. All animals (regardless of sex, dose, and PAV type) could respond in the active nose-poke, which produced the stimulus alone, or in the inactive nose-poke. Both males and females dose-dependently acquired nose-poke responding (active>inactive) after paired PAV, but not after random PAV. Therefore, the stimulus was a conditioned reinforcer. We identified three sex differences. First, only females acquired responding after paired PAV with 32.0 μg/kg remifentanil. Second, using a progressive ratio schedule for ACQ, both sexes acquired responding, but females made significantly more active responses. Third, when a single session of PAV was conducted, only males acquired responding. Thus, rats' sex interacts with pharmacological and environmental factors to determine opioid-conditioned reinforcement.

The power of associative learning and the ontogeny of optimal behaviour

Behaving efficiently (optimally or near-optimally) is central to animals' adaptation to their environment. Much evolutionary biology assumes, implicitly or explicitly, that optimal behavioural strategies are genetically inherited, yet the behaviour of many animals depends crucially on learning. The question of how learning contributes to optimal behaviour is largely open. Here we propose an associative learning model that can learn optimal behaviour in a wide variety of ecologically relevant circumstances. The model learns through chaining, a term introduced by Skinner to indicate learning of behaviour sequences by linking together shorter sequences or single behaviours. Our model formalizes the concept of conditioned reinforcement (the learning process that underlies chaining) and is closely related to optimization algorithms from machine learning. Our analysis dispels the common belief that associative learning is too limited to produce 'intelligent' behaviour such as tool use, social learning, self-control or expectations of the future. Furthermore, the model readily accounts for both instinctual and learned aspects of behaviour, clarifying how genetic evolution and individual learning complement each other, and bridging a long-standing divide between ethology and psychology. We conclude that associative learning, supported by genetic predispositions and including the oft-neglected phenomenon of conditioned reinforcement, may suffice to explain the ontogeny of optimal behaviour in most, if not all, non-human animals. Our results establish associative learning as a more powerful optimizing mechanism than acknowledged by current opinion.

The Origins and Organization of Vertebrate Pavlovian Conditioning

Pavlovian conditioning is the process by which we learn relationships between stimuli and thus constitutes a basic building block for how the brain constructs representations of the world. We first review the major concepts of Pavlovian conditioning and point out many of the pervasive misunderstandings about just what conditioning is. This brings us to a modern redefinition of conditioning as the process whereby experience with a conditional relationship between stimuli bestows these stimuli with the ability to promote adaptive behavior patterns that did not occur before the experience. Working from this framework, we provide an in-depth analysis of two examples, fear conditioning and food-based appetitive conditioning, which include a description of the only partially overlapping neural circuitry of each. We also describe how these circuits promote the basic characteristics that define Pavlovian conditioning, such as error-correction-driven regulation of learning.

A Unifying Probabilistic View of Associative Learning

Two important ideas about associative learning have emerged in recent decades: (1) Animals are Bayesian learners, tracking their uncertainty about associations; and (2) animals acquire long-term reward predictions through reinforcement learning. Both of these ideas are normative, in the sense that they are derived from rational design principles. They are also descriptive, capturing a wide range of empirical phenomena that troubled earlier theories. This article describes a unifying framework encompassing Bayesian and reinforcement learning theories of associative learning. Each perspective captures a different aspect of associative learning, and their synthesis offers insight into phenomena that neither perspective can explain on its own.

Implications of learning theory for developing programs to decrease overeating

Childhood obesity is associated with medical and psychological comorbidities, and interventions targeting overeating could be pragmatic and have a significant impact on weight. Calorically dense foods are easily available, variable, and tasty which allows for effective opportunities to learn to associate behaviors and cues in the environment with food through fundamental conditioning processes, resulting in measurable psychological and physiological food cue reactivity in vulnerable children. Basic research suggests that initial learning is difficult to erase, and that it is vulnerable to a number of phenomena that will allow the original learning to re-emerge after it is suppressed or replaced. These processes may help explain why it may be difficult to change food cue reactivity and overeating over the long term. Extinction theory may be used to develop effective cue-exposure treatments to decrease food cue reactivity through inhibitory learning, although these processes are complex and require an integral understanding of the theory and individual differences. Additionally, learning theory can be used to develop other interventions that may prove to be useful. Through an integration of learning theory, basic and translational research, it may be possible to develop interventions that can decrease the urges to overeat, and improve the weight status of children.