Separation of time-based and trial-based accounts of the partial reinforcement extinction effect

Partial reinforcement effects on acquisition and extinction of a conditioned taste aversion

Four experiments with rat subjects asked whether a partial reinforcement extinction effect (PREE) occurs in taste aversion learning. The question has received little attention in the literature, and to our knowledge no taste aversion experiment has previously demonstrated a PREE. In each of the present experiments, experimental groups received a taste mixed in drinking water for 20 min; such taste exposures were sometimes paired with a lithium chloride (LiCl) injection and sometimes not. Control groups received only taste-LiCl pairings. There was evidence that each reinforced and non-reinforced trial produced increments and decrements in aversion strength (respectively), and trials mattered more than accumulated time during the conditioned stimulus and during the background (as emphasized in time-accumulation models like those of Gallistel and Gibbon, Psychological Review, 107, 289-344, 2000, and Gibbon and Balsam, Autoshaping and conditioning theory, Academic Press, New York, pp. 219-235, 1981). In addition, a partial reinforcement extinction effect was observed when there was a relatively large number of conditioning trials. The results extend our understanding of extinction in taste aversion learning and provide more evidence that aversion learning might follow rules that are qualitatively similar to those of other forms of learning.

EXPRESS: Determinants of Extinction in a Streamed Trial Procedure

The strength of an association between a cue and its outcome is influenced by both the probability of the outcome given the cue and the probability of the outcome in the absence of the cue. Once an association has been formed, extinction is the procedure for reducing responding indicative of the association by repeated presentation of the cue without the outcome. The present experiments tested whether cumulative frequency and/or cumulative duration of these events affects associative extinction in a streamed trial extinction procedure with human participants. Experiment 1 assessed the effects of parametric manipulations of the frequency and duration of either the cue by itself or cue-outcome coabsence. In Experiment 1, participants proved relatively insensitive to manipulation of the event's duration. In contrast, judgments of the association by participants decreased when the frequency of cue-alone events was increased, even when the durations of those events were decreased so that cumulative exposure to the cue was equated. No effect of either the duration or the frequency of cue-outcome coabsence was observed. Experiment 2 demonstrated that the effect of cue-alone (i.e., extinction trial) frequency generalizes across a wide range of parameters for initial acquisition achieved by cue-outcome pairings. Experiment 3 tested for an interaction between event duration during initial learning and event duration during extinction. Collectively, these results indicate that the cumulative frequency, and not the cumulative duration, of extinction trials as well as the duration of the cue-outcome coabsences between extinction trials controls the effectiveness of an extinction procedure.

BEHAVIORAL AND NEUROBIOLOGICAL MECHANISMS OF PAVLOVIAN AND INSTRUMENTAL EXTINCTION LEARNING

This article reviews the behavioral neuroscience of extinction, the phenomenon in which a behavior that has been acquired through Pavlovian or instrumental (operant) learning decreases in strength when the outcome that reinforced it is removed. Behavioral research indicates that neither Pavlovian nor operant extinction depends substantially on erasure of the original learning but instead depends on new inhibitory learning that is primarily expressed in the context in which it is learned, as exemplified by the renewal effect. Although the nature of the inhibition may differ in Pavlovian and operant extinction, in either case the decline in responding may depend on both generalization decrement and the correction of prediction error. At the neural level, Pavlovian extinction requires a tripartite neural circuit involving the amygdala, prefrontal cortex, and hippocampus. Synaptic plasticity in the amygdala is essential for extinction learning, and prefrontal cortical inhibition of amygdala neurons encoding fear memories is involved in extinction retrieval. Hippocampal-prefrontal circuits mediate fear relapse phenomena, including renewal. Instrumental extinction involves distinct ensembles in corticostriatal, striatopallidal, and striatohypothalamic circuits as well as their thalamic returns for inhibitory (extinction) and excitatory (renewal and other relapse phenomena) control over operant responding. The field has made significant progress in recent decades, although a fully integrated biobehavioral understanding still awaits.

A systems-neuroscience model of phasic dopamine

We describe a neurobiologically informed computational model of phasic dopamine signaling to account for a wide range of findings, including many considered inconsistent with the simple reward prediction error (RPE) formalism. The central feature of this PVLV framework is a distinction between a primary value (PV) system for anticipating primary rewards (Unconditioned Stimuli [USs]), and a learned value (LV) system for learning about stimuli associated with such rewards (CSs). The LV system represents the amygdala, which drives phasic bursting in midbrain dopamine areas, while the PV system represents the ventral striatum, which drives shunting inhibition of dopamine for expected USs (via direct inhibitory projections) and phasic pausing for expected USs (via the lateral habenula). Our model accounts for data supporting the separability of these systems, including individual differences in CS-based (sign-tracking) versus US-based learning (goal-tracking). Both systems use competing opponent-processing pathways representing evidence for and against specific USs, which can explain data dissociating the processes involved in acquisition versus extinction conditioning. Further, opponent processing proved critical in accounting for the full range of conditioned inhibition phenomena, and the closely related paradigm of second-order conditioning. Finally, we show how additional separable pathways representing aversive USs, largely mirroring those for appetitive USs, also have important differences from the positive valence case, allowing the model to account for several important phenomena in aversive conditioning. Overall, accounting for all of these phenomena strongly constrains the model, thus providing a well-validated framework for understanding phasic dopamine signaling. (PsycInfo Database Record (c) 2020 APA, all rights reserved).

Negative reinforcement rate and persistent avoidance following response-prevention extinction

Persistent avoidance may be influenced by prior negative reinforcement rate (i.e., how effective the response is at controlling threat). In clinical settings, the effectiveness of extinction-based methods for treating anxiety-related avoidance may be impacted by prior reinforcement rate. Here, we conducted a laboratory-based treatment study to investigate the persistence of avoidance following response-prevention extinction (RPE) when prior avoidance had been differentially effective at cancelling shock. Participants in three negative reinforcement rate groups (100%, 50%, and 0%) completed a validated avoidance conditioning paradigm involving Pavlovian fear extinction, RPE, and re-extinction phases. It was hypothesised that partially reinforced avoidance would lead to diminished resistance to fear extinction following response prevention, compared to continuously- or never-reinforced avoidance. Persistent avoidance was related to prior negative reinforcement rate, with higher rates more resistant to extinction. These findings illustrate the role of reinforcement rate in the persistence of avoidance and may aid understanding of treatment relapse.

Effects of reward magnitude and training frequency on the learning rates and memory retention of the Port Jackson shark Heterodontus portusjacksoni

The development of adaptive responses to novel situations via learning has been demonstrated in a wide variety of animal taxa. However, knowledge on the learning abilities of one of the oldest extant vertebrate groups, Chondrichthyes, remains limited. With the increasing interest in global wildlife tourism and shark feeding operations, it is important to understand the capacities of these animals to form associations between human activities and food. We used an operant conditioning regime with a simple spatial cognitive task to investigate the effects of reinforcement frequency and reward magnitude on the learning performance and memory retention of Port Jackson sharks (Heterodontus portusjacksoni). Twenty-four Port Jackson sharks were assigned one of four treatments differing in reward magnitude and reinforcement frequency (large magnitude-high frequency; large magnitude-low frequency; small magnitude-high frequency; small magnitude-low frequency). The sharks were trained over a 21-day period to compare the number of days that it took to learn to pass an assigned door to feed. Sharks trained at a high reinforcement frequency demonstrated faster learning rates and a higher number of passes through the correct door at the end of the trials, while reward magnitude had limited effects on learning rate. This suggests that a reduction in reinforcement frequency during tourism-related feeding operations is likely to be more effective in reducing the risk of sharks making associations with food than limiting the amount of food provided.

Estradiol Modulates Neural and Behavioral Arousal in Women With Posttraumatic Stress Disorder During a Fear Learning and Extinction Task

BACKGROUND

Fear responding in posttraumatic stress disorder (PTSD) is sexually heterogeneous and varies with hormonal fluctuations throughout the menstrual cycle. While research suggests that estrogen levels affect PTSD symptoms among women, there is a dearth of research on modulatory effects of estrogen on fear responding among women with PTSD, and neural outcome measures are lacking.

METHODS

A sample of 42 women with PTSD underwent 2 consecutive alternating blocks of fear conditioning and extinction training, during which a CS+ conditioned stimulus, but not a CS-, predicted the occurrence of an electric shock in an acquisition context but not in an extinction context. Assayed saliva determined estradiol levels. Skin conductance response and whole-brain voxelwise activity during functional magnetic resonance imaging were outcome variables in linear mixed-effects models, with estradiol level, PTSD severity, and task contrasts as predictors.

RESULTS

Skin conductance response exhibited a significant estradiol × PTSD severity × habituation interaction (t = 3.180, p = .002), such that PTSD severity was correlated with increased arousal responding between training blocks among women with lower estradiol (t = -3.985, p < .001) but not higher estradiol (t = 0.550, p = .583). Voxelwise activity also demonstrated an identical three-way interaction within dorsomedial prefrontal cortex and anterior insula clusters. The skin conductance response and imaging interactions between PTSD severity and estradiol were not specific to conditioned stimulus type or context.

CONCLUSIONS

Estradiol moderated the relationship between PTSD severity and arousal response habituation between fear conditioning and extinction training sessions, such that high estradiol protected against the negative impact of severe PTSD symptoms on fear habituation. These findings suggest that estrogen enhances habituation among women with severe PTSD, potentially influencing the efficacy of extinction-based therapies.

Making translation work: Harmonizing cross-species methodology in the behavioural neuroscience of Pavlovian fear conditioning

Translational neuroscience bridges insights from specific mechanisms in rodents to complex functions in humans and is key to advance our general understanding of central nervous function. A prime example of translational research is the study of cross-species mechanisms that underlie responding to learned threats, by employing Pavlovian fear conditioning protocols in rodents and humans. Hitherto, evidence for (and critique of) these cross-species comparisons in fear conditioning research was based on theoretical viewpoints. Here, we provide a perspective to substantiate these theoretical concepts with empirical considerations of cross-species methodology. This meta-research perspective is expected to foster cross-species comparability and reproducibility to ultimately facilitate successful transfer of results from basic science into clinical applications.

Rewards interact with repetition-dependent learning to enhance long-term retention of motor memories

The combination of behavioral experiences that enhance long-term retention remains largely unknown. Informed by neurophysiological lines of work, this study tested the hypothesis that performance-contingent monetary rewards potentiate repetition-dependent forms of learning, as induced by extensive practice at asymptote, to enhance long-term retention of motor memories. To this end, six groups of 14 participants (n = 84) acquired novel motor behaviors by adapting to a gradual visuomotor rotation while these factors were manipulated. Retention was assessed 24 h later. While all groups similarly acquired the novel motor behaviors, results from the retention session revealed an interaction indicating that rewards enhanced long-term retention, but only when practice was extended to asymptote. Specifically, the interaction indicated that this effect selectively occurred when rewards were intermittently available (i.e., 50%), but not when they were absent (i.e., 0%) or continuously available (i.e., 100%) during acquisition. This suggests that the influence of rewards on extensive practice and long-term retention is nonlinear, as continuous rewards did not further enhance retention as compared with intermittent rewards. One possibility is that rewards' intermittent availability allows to maintain their subjective value during acquisition, which may be key to potentiate long-term retention.

Translational tests involving non-reward: methodological considerations

This review is concerned with methods for assessing the processing of unrewarded responses in experimental animals and the mechanisms underlying performance of these tasks. A number of clinical populations, including Parkinson's disease, depression, compulsive disorders, and schizophrenia demonstrate either abnormal processing or learning from non-rewarded responses in laboratory-based reinforcement learning tasks. These effects are hypothesized to result from disturbances in modulatory neurotransmitter systems, including dopamine and serotonin. Parallel work in experimental animals has revealed consistent behavioral patterns associated with non-reward and, consistent with the human literature, modulatory roles for specific neurotransmitters. Classical tests involving an important reward omission component include appetitive extinction, ratio schedules of responding, reversal learning, and delay and probability discounting procedures. In addition, innovative behavioral tests have recently been developed leverage probabilistic feedback to specifically assay accommodation of, and learning from, non-rewarded responses. These procedures will be described and reviewed with discussion of the behavioral and neural determinants of performance. A final section focusses specifically on the benefits of trial-by-trial analysis of responding during such tasks, and the implications of such analyses for the translation of findings to clinical studies.

Rescaling of temporal expectations during extinction

Previous research suggests that extinction learning is temporally specific. Changing the conditioned stimulus (CS) duration between training and extinction can facilitate the loss of the conditioned response (CR) within the extinction session but impairs long-term retention of extinction. In 2 experiments using conditioned magazine approach with rats, we examined the relation between temporal specificity of extinction and CR timing. In Experiment 1, rats were trained on a 12-s, fixed CS-unconditional stimulus interval and then extinguished with CS presentations that were 6, 12, or 24 s in duration. The design of Experiment 2 was the same except rats were trained using partial rather than continuous reinforcement. In both experiments, extending the CS duration in extinction facilitated the diminution of CRs during the extinction session, but shortening the CS duration failed to slow extinction. In addition, extending (but not shortening) the CS duration caused temporal rescaling of the CR, in that the peak CR rate migrated later into the trial over the course of extinction training. This migration partially accounted for the faster loss of the CR when the CS duration was extended. Results are incompatible with the hypothesis that extinction is driven by cumulative CS exposure and suggest that temporally extended nonreinforced CS exposure reduces conditioned responding via temporal displacement rather than through extinction per se. (PsycINFO Database Record

Bayesian methods for addressing long-standing problems in associative learning: The case of PREE

Most associative models typically assume that learning can be understood as a gradual change in associative strength that captures the situation into one single parameter, or representational state. We will call this view single-state learning. However, there is ample evidence showing that under many circumstances different relationships that share features can be learned independently, and animals can quickly switch between expressing one or another. We will call this multiple-state learning. Theoretically, it is understudied because it needs a different data analysis approach from those usually employed. In this article, we present a Bayesian model of the Partial Reinforcement Extinction Effect (PREE) that can test the predictions of the multiple-state view. This implies estimating the moment of change in the responses (from the acquisition to the extinction performance), both at the individual and group levels. We used this model to analyze data from a PREE experiment with three levels of reinforcement during acquisition (100%, 75% and 50%). We found differences in the estimated moment of switch between states during extinction, so that it was delayed after leaner partial reinforcement schedules. The finding is compatible with the multiple-state view. It is the first time, to our knowledge, that the predictions from the multiple-state view are tested directly. The article also aims to show the benefits that Bayesian methods can bring to the associative learning field.

Factors that influence the persistence and relapse of discriminated behavior chains

Behavior chains are composed of sequences of behaviors that minimally include procurement and then consumption. This review surveys recent research from this laboratory that has examined the properties of discriminated heterogeneous behavior chains. In contrast to another review (Thrailkill and Bouton, 2016a), it discusses work examining what makes chained behavior persistent, and what makes it relapse. Results suggest that responses in a discriminated heterogeneous behavior chain may become associated, so that extinction of either one reduces the strength of the other. Evidence also suggests that the goal of the first (procurement) response may be the next (consumption) response (rather than the upcoming discriminative stimulus, a putative conditioned reinforcer, or the primary reinforcer at the end of the chain). Further studies suggest that methods that promote generalization across acquisition and extinction (partial reinforcement and delivery of noncontingent reinforcers during extinction) lead to greater persistence of the procurement response. A third set of studies analyzed the contextual control and relapse of chained behaviors. The context controls both the acquisition and extinction of chained behaviors. In addition, a separately-extinguished consumption response is renewed when returned to the context of the chain. The research expands our general understanding of the learning processes that govern instrumental behavior as well as our understanding of chains.

Understanding the psychology of mobile gambling: A behavioural synthesis

This manuscript reviews the extant literature on key issues related to mobile gambling and considers whether the potential risks of harm emerging from this platform are driven by pre‐existing comorbidities or by psychological processes unique to mobile gambling. We propose an account based on associative learning that suggests this form of gambling is likely to show distinctive features compared with other gambling technologies. Smartphones are a rapidly growing platform on which individuals can gamble using specifically designed applications, adapted websites or text messaging. This review considers how mobile phone use interacts with psychological processes relevant to gambling, the games users are likely to play on smartphones, and the interactions afforded by smartphones. Our interpretation of the evidence is that the schedules of reinforcement found in gambling interact with the ways in which people tend to use smartphones that may expedite the acquisition of maladaptive learned behaviours such as problem gambling. This account is consistent with existing theories and frameworks of problem gambling and has relevance to other forms of mobile phone use.

Increasing the persistence of a heterogeneous behavior chain: Studies of extinction in a rat model of search behavior of working dogs

Dogs trained to search for contraband perform a chain of behavior in which they first search for a target and then make a separate response that indicates to the trainer that they have found one. The dogs often conduct multiple searches without encountering a target and receiving the reinforcer (i.e., no contraband is present). Understanding extinction (i.e., the decline in work rate when reinforcers are no longer encountered) may assist in training dogs to work in conditions where targets are rare. We therefore trained rats on a search-target behavior chain modeled on the search behavior of working dogs. A discriminative stimulus signaled that a search response (e.g., chain pull) led to a second stimulus that set the occasion for a target response (e.g., lever press) that was reinforced by a food pellet. In Experiment 1 training with longer search durations and intermittent (partial) reinforcement of searching (i.e. some trials had no target present) both led to more persistent search responding in extinction. The loss of search behavior in extinction was primarily dependent on the number of non-reinforced searches rather than time searching without reinforcement. In Experiments 2 and 3, delivery of non-contingent reinforcers during extinction increased search persistence provided they had also been presented during training. Thus, results with rats suggest that the persistence of working dog performance (or chained behavior generally) may be improved by training with partial reinforcement of searching and non-contingent reinforcement during both training and work (extinction).

Why are Some Games More Addictive than Others: The Effects of Timing and Payoff on Perseverance in a Slot Machine Game

Manipulating different behavioral characteristics of gambling games can potentially affect the extent to which individuals persevere at gambling, and their transition to problematic behaviors. This has potential impact for mobile gambling technologies and responsible gambling interventions. Two laboratory models pertinent to this are the partial reinforcement extinction effect (PREE) and the trial spacing effect. Both of these might speed up or delay the acquisition and extinction of conditioned behavior. We report an experiment that manipulated the rate of reinforcement and inter trial interval (ITI) on a simulated slot machine where participants were given the choice between gambling and skipping on each trial, before perseverative gambling was measured in extinction, followed by measurements of the illusion of control, depression and impulsivity. We hypothesized that longer ITI’s in conjunction with the low rates of reinforcement observed in gambling would lead to greater perseverance. We further hypothesized, given that timing is known to be important in displaying illusory control and potentially in persevering in gambling, that prior exposure to longer intervals might affect illusions of control. An interaction between ITI and rate of reinforcement was observed, as low reinforced gamblers with a long ITI gambled for longer. Respondents also displayed extinction and a PREE. Gamblers exposed to a higher rate of reinforcement gambled for longer in acquisition. Impulsivity was associated with extended perseverance in extinction, and more depressed gamblers in the high reinforcement short ITI group persevered for longer. Performance in the contingency judgment failed to support the second hypothesis: the only significant contrast observed was that participants became better calibrated as the task progressed.